Nov 01 20:01:59.285668 ip-10-0-128-78 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:01:59.666218 ip-10-0-128-78 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:01:59.666218 ip-10-0-128-78 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:01:59.666218 ip-10-0-128-78 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:01:59.666218 ip-10-0-128-78 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:01:59.670897 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670765 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.670897 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670891 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.670897 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670897 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.670897 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670902 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.670897 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670907 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670912 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670917 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670921 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670924 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670928 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670932 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670935 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670938 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670942 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670946 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670950 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670954 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670957 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670961 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670964 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670970 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670974 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670978 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670982 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.671237 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670986 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670990 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670994 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.670998 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671002 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671005 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671009 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671012 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671016 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671020 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671024 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671028 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671032 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671036 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671041 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671045 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671050 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671054 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671058 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.672043 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671063 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671067 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671074 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671079 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671084 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671088 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671092 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671097 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671102 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671110 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671115 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671121 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671126 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671131 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671135 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671139 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671143 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671148 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671152 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.672570 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671156 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671160 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671165 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671169 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671173 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671178 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671182 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671186 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671190 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671195 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671200 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671205 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671209 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671213 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671218 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671222 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671226 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671230 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671233 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.673053 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671238 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671242 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671246 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671250 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671254 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671259 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671264 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671269 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671273 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671278 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671282 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671286 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671290 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671294 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671298 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671302 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671307 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671311 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671316 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.673533 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671320 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.674014 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671324 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.674014 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671348 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.674014 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671353 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.674014 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671357 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.674014 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.671362 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.675068 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675044 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.675068 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675065 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.675068 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675070 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675074 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675079 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675082 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675089 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675095 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675099 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675102 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675106 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675111 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675115 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675119 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675123 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675126 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675130 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675134 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675138 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675142 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675146 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.675264 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675150 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675154 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675158 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675161 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675165 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675169 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675173 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675177 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675181 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675186 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675190 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675196 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675201 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675205 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675209 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675215 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675219 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675223 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675228 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675232 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.676070 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675236 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675240 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675245 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675249 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675254 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675258 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675262 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675266 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675270 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675275 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675279 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675284 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675288 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675292 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675296 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675300 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675304 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675309 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675313 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675317 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.676697 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675321 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675343 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675349 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675353 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675358 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675363 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675367 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675373 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675380 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675384 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675388 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675393 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675397 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675401 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675406 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675410 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675417 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675425 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675430 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.677178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675434 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675439 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675443 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675448 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675452 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675456 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675460 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675464 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675468 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675472 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675476 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675481 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675485 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675489 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675493 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675498 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675502 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675507 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675512 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675517 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.677660 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675522 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675527 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675531 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675535 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675539 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.675546 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676466 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676481 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676485 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676488 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676492 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676495 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676498 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676500 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676503 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676506 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676508 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676511 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676513 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676517 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.678146 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676519 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676522 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676525 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676527 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676530 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676532 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676535 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676537 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676540 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676544 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676548 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676550 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676553 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676555 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676558 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676560 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676563 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676566 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676570 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.678639 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676574 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676576 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676580 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676584 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676587 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676590 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676592 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676595 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676598 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676600 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676603 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676606 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676608 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676611 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676613 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676616 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676618 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676621 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676624 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676627 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.679178 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676630 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676632 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676635 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676637 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676640 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676642 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676645 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676647 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676650 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676652 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676655 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676658 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676661 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676665 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676668 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676670 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676680 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676683 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676685 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676688 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.679713 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676691 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676693 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676696 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676698 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676701 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676704 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676707 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676710 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676712 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676715 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676717 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676720 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676722 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676725 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676728 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676730 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676734 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676737 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676739 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.680175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676741 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676744 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676746 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676749 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676752 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676755 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676757 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676761 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676764 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676766 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676769 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676771 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676773 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.676776 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677840 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677851 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677860 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677865 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677870 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677873 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677878 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:01:59.680637 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677883 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677886 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677889 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677893 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677896 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677899 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677902 2573 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677905 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677908 2573 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677911 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677914 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677918 2573 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677921 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677924 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677928 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677933 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677937 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677941 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677944 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677948 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677951 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677954 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677957 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677961 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:01:59.681166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677964 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677969 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677972 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677975 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677978 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677981 2573 flags.go:64] FLAG: --enable-server="true" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677984 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677989 2573 flags.go:64] FLAG: --event-burst="100" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677992 2573 flags.go:64] FLAG: --event-qps="50" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677995 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.677999 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678002 2573 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678007 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678014 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678017 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678021 2573 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678024 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678027 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678029 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678032 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678035 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678038 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678041 2573 flags.go:64] FLAG: --feature-gates="" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678046 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678049 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:01:59.681785 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678052 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678055 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678058 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678062 2573 flags.go:64] FLAG: --help="false" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678065 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678068 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678071 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678074 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678079 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678082 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678085 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678088 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678091 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678094 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678098 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678101 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678104 2573 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678107 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678110 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678113 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678117 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678120 2573 flags.go:64] FLAG: --lock-file="" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678124 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678127 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:01:59.682381 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678129 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678136 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678139 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678142 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678145 2573 flags.go:64] FLAG: --logging-format="text" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678148 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678152 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678155 2573 flags.go:64] FLAG: --manifest-url="" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678159 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678164 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678167 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678171 2573 flags.go:64] FLAG: --max-pods="110" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678174 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678177 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678180 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678183 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678186 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678189 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678192 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678202 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678205 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678208 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678211 2573 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678214 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678217 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:01:59.682964 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678220 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678223 2573 flags.go:64] FLAG: --port="10250" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678226 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678229 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-005c246d98ad74763" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678234 2573 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678237 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678240 2573 flags.go:64] FLAG: --register-node="true" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678243 2573 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678247 2573 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678250 2573 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678253 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678256 2573 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678260 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678264 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678267 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678270 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678273 2573 flags.go:64] FLAG: --runonce="false" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678276 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678279 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678282 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678285 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678288 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678291 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678295 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678298 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678301 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:01:59.683587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678304 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678307 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678310 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678313 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678316 2573 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678318 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678324 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678339 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678343 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678347 2573 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678350 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678353 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678356 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678358 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678362 2573 flags.go:64] FLAG: --v="2" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678367 2573 flags.go:64] FLAG: --version="false" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678371 2573 flags.go:64] FLAG: --vmodule="" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678375 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678379 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678528 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678533 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678536 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678539 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678542 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.684209 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678544 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678547 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678550 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678553 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678556 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678560 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678564 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678567 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678570 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678573 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678576 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678581 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678583 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678586 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678589 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678591 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678594 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678596 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678599 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.684838 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678602 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678605 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678608 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678610 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678613 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678616 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678618 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678621 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678623 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678626 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678628 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678631 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678634 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678636 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678639 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678642 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678645 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678648 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678651 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.685311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678654 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678656 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678659 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678661 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678664 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678669 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678672 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678674 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678677 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678679 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678682 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678685 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678687 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678690 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678692 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678695 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678697 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678699 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678702 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.685793 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678704 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678707 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678709 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678712 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678714 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678716 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678720 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678724 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678727 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678729 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678732 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678734 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678737 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678740 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678742 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678745 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678747 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678750 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678754 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678757 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.686240 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678760 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678762 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678765 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678767 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678771 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678773 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678776 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678778 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678781 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678783 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678786 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678789 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678791 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678794 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678796 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678799 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678801 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678804 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678806 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678809 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.686735 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678811 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678814 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678816 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678819 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.678849 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678888 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678892 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678895 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678898 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678901 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678905 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678907 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678910 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678912 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678915 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678917 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.687207 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678920 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678924 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678927 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678930 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678932 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678935 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678937 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678940 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678942 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678945 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678948 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678950 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678953 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678955 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678958 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678960 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678963 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678965 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678968 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678970 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.687615 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678973 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678976 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678979 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678981 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678984 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678986 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678990 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678992 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.678996 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679001 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679005 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679007 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679010 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679014 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679017 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679019 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679022 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679024 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679027 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.688078 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679030 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679032 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679035 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679038 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679040 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679043 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679045 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679048 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679050 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679053 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679056 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679058 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679061 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679063 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679066 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679068 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679072 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679075 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679077 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679081 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.688581 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679084 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679087 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679089 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679092 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679094 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679097 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679101 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679103 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679106 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679108 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679111 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679114 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679117 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679120 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679122 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679125 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679128 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679130 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679133 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.689062 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679135 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679138 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679140 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679143 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679145 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679148 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679150 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679153 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679156 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679160 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679164 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679166 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679170 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679172 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679175 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679178 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.679180 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.687639 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.687657 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.688362 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:01:59.689521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.688372 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688437 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688443 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688447 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688450 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688452 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688455 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688457 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688460 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688463 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688466 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688469 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688471 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688474 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688476 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688479 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688481 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688483 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688486 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688489 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.690300 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688491 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688493 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688496 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688498 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688501 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688504 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688508 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688510 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688513 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688515 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688519 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688522 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688525 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688528 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688530 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688533 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688536 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688538 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688541 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688544 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.691036 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688546 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688549 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688551 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688554 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688556 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688559 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688563 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688568 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688571 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688574 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688576 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688579 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688582 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688585 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688588 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688591 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688593 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688596 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688599 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688601 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.691540 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688604 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688607 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688609 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688612 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688615 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688618 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688621 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688623 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688626 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688629 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688631 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688634 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688636 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688639 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688641 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688644 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688646 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688649 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688651 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688653 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.692012 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688656 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688658 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688661 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688663 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688665 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688668 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688680 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688684 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688687 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688689 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688694 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688698 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688701 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688704 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688706 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688710 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688713 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688716 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688718 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.692501 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688721 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688724 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688726 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688729 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688731 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688734 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688736 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688739 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.688772 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688918 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688923 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688926 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688929 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688932 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688934 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.692952 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688937 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688939 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688942 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688944 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688947 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688949 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688951 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688955 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688957 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688960 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688963 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688965 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688967 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688970 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688973 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688976 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688980 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688983 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688986 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688988 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.693310 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688990 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688993 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688995 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.688998 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689000 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689003 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689005 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689008 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689010 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689013 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689015 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689018 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689020 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689023 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689025 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689028 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689030 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689033 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689035 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.693806 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689038 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689040 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689043 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689045 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689047 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689050 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689052 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689055 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689058 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689061 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689065 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689068 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689071 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689074 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689077 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689080 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689083 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689085 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689088 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689090 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.694261 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689092 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689095 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689097 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689100 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689102 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689104 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689106 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689109 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689111 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689114 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689118 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689121 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689124 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689127 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689129 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689132 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689134 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689137 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689139 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.694737 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689142 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689144 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689147 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689150 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689152 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689155 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689157 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689160 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689163 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689167 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689169 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689171 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689174 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689177 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689179 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689182 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689184 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689187 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689189 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689192 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.695175 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689194 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689197 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.689221 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689256 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689262 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689265 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689268 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689271 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689273 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689276 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689279 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689281 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689284 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689286 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689289 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689291 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:59.695673 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689295 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689298 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689300 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689303 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689305 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689309 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689311 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689314 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689316 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689319 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689321 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689323 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689326 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689346 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689349 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689352 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689354 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689357 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689360 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689362 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:59.696056 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689366 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689369 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689372 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689374 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689377 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689379 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689382 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689384 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689387 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689390 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689392 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689395 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689398 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689400 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689403 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689406 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689408 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689413 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689416 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:59.696646 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689419 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689423 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689425 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689428 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689431 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689433 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689436 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689438 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689440 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689443 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689445 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689448 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689451 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689455 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689458 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689460 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689463 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689465 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689468 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689471 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:59.697118 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689473 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689476 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689479 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689481 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689483 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689486 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689488 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689491 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689494 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689497 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689500 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689502 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689505 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689509 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689512 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689516 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689518 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689521 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689523 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:59.697605 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689526 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689529 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689531 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689534 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689537 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689540 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689544 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689546 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689549 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689552 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689554 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689557 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689559 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689562 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:01:59.689564 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.693184 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.695535 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.696458 2573 server.go:1014] "Starting client certificate rotation" Nov 01 20:01:59.698045 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.696557 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:01:59.698510 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.696594 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:01:59.698510 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.698408 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:01:59.699595 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.699578 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:01:59.709484 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.709466 2573 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:01:59.713170 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.713152 2573 log.go:25] "Validated CRI v1 image API" Nov 01 20:01:59.714065 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.714034 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:01:59.719083 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.719051 2573 fs.go:135] Filesystem UUIDs: map[52e88fd5-fee8-4410-b6e7-195d5680e98a:/dev/nvme0n1p4 678924c3-63db-47a3-a938-c326d17f0743:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 01 20:01:59.719201 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.719083 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 20:01:59.725291 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.725142 2573 manager.go:217] Machine: {Timestamp:2026-11-01 20:01:59.723673076 +0000 UTC m=+0.333854013 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084724 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff91cd15200a8acd4b0cc08e09498 SystemUUID:ec2ff91c-d152-00a8-acd4-b0cc08e09498 BootID:4237194a-4dc3-4052-be14-7bbb92af4feb Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:39:3a:64:78:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:39:3a:64:78:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:b5:9b:41:7e:fb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:01:59.725291 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.725275 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:01:59.725506 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.725428 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:01:59.729052 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.729015 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:01:59.729233 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.729055 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-78.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:01:59.729274 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.729236 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:01:59.729274 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.729244 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:01:59.729355 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.729281 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:01:59.729355 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.729294 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:01:59.730513 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.730501 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:01:59.734573 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.734559 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:01:59.734718 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.734708 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:01:59.735502 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.735490 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:01:59.735534 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.735508 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:01:59.735534 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.735518 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:01:59.735534 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.735527 2573 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:01:59.735661 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.735538 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:01:59.735902 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.735887 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:01:59.739466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.738627 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:01:59.745500 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.745485 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:01:59.746446 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746435 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:01:59.746485 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746465 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:01:59.746626 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746615 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:01:59.746659 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746631 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:01:59.746659 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746638 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:01:59.746659 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746644 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:01:59.746659 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746649 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:01:59.746659 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.746656 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:01:59.747440 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.747432 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:01:59.747475 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.747441 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:01:59.747475 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.747448 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:01:59.747475 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.747454 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:01:59.747475 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.747463 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:01:59.747475 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.747472 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:01:59.749022 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.749011 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:01:59.749022 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.749022 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:01:59.753032 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.753016 2573 server.go:1263] "Started kubelet" Nov 01 20:01:59.753142 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.753095 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:01:59.753216 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.753129 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:01:59.753250 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.753241 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:01:59.753973 ip-10-0-128-78 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:01:59.755184 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.755167 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:01:59.757440 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.757421 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:01:59.759660 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:01:59.759641 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 20:01:59.762950 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.762929 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:01:59.763540 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.763524 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:01:59.763635 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:01:59.763609 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" Nov 01 20:01:59.764364 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764321 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:01:59.764364 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764352 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:01:59.764527 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764320 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:01:59.764527 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764440 2573 factory.go:55] Registering systemd factory Nov 01 20:01:59.764527 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764459 2573 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:01:59.764661 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764533 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:01:59.764661 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764544 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:01:59.764782 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764757 2573 factory.go:153] Registering CRI-O factory Nov 01 20:01:59.764830 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764787 2573 factory.go:223] Registration of the crio container factory successfully Nov 01 20:01:59.764874 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764846 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:01:59.764874 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764869 2573 factory.go:103] Registering Raw factory Nov 01 20:01:59.764954 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.764885 2573 manager.go:1199] Started watching for new ooms in manager Nov 01 20:01:59.766324 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.766026 2573 manager.go:319] Starting recovery of all containers Nov 01 20:01:59.775190 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.775163 2573 manager.go:324] Recovery completed Nov 01 20:01:59.781392 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.781371 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:59.783199 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.783163 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:59.783288 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.783219 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:59.783288 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.783239 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:59.783932 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.783916 2573 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:01:59.783932 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.783930 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:01:59.784036 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.783935 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:59.784036 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.783947 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:01:59.784672 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.784656 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:59.784733 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.784692 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:59.784733 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.784703 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:59.786064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.786052 2573 policy_none.go:50] "Start" Nov 01 20:01:59.786110 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.786068 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:01:59.786110 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.786078 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:01:59.787961 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.787949 2573 policy_none.go:44] "Start" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.839560 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.846360 2573 manager.go:341] "Starting Device Plugin manager" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:01:59.846400 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.846414 2573 server.go:87] "Starting device plugin registration server" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.846744 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.846756 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.846854 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.846971 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.846981 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:01:59.847525 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:01:59.858731 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:01:59.847573 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" Nov 01 20:01:59.861184 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.859419 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:01:59.861184 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.860860 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:01:59.861184 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.860892 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:01:59.861184 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.860917 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:01:59.861184 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:01:59.860959 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 20:01:59.946943 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.946860 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:59.947894 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.947870 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:59.948013 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.947905 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:59.948013 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.947922 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:59.948013 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.947948 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:01:59.962241 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.962212 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal"] Nov 01 20:01:59.962319 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.962279 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:59.965724 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.965704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:01:59.965779 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.965734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:01:59.966413 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.966390 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:59.966413 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.966419 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:59.966575 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.966429 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:59.968578 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.968565 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:59.968753 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.968731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:01:59.968797 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.968781 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:59.969410 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.969391 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:59.969521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.969420 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:59.969521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.969434 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:59.969521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.969468 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:59.969521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.969489 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:59.969521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.969499 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:59.971497 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.971478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:01:59.971593 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.971516 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:59.972288 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.972270 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:59.972414 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.972302 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:59.972414 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.972315 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:59.974963 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.974946 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:01:59.975047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:01:59.975027 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5qslt" Nov 01 20:02:00.007072 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:00.007046 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" node="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.009767 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:00.009747 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" node="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.047608 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.047588 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:02:00.065931 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.065901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.065931 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.065934 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.066156 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.065957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c04d0eff16222e7baef223e711863fea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"c04d0eff16222e7baef223e711863fea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.066156 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.066032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.066156 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.066048 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.096082 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.096052 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:02:00.096174 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.096102 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:02:00.146571 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.146547 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:02:00.147462 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.147443 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.147553 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:00.147476 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-78.us-west-1.compute.internal\": node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" Nov 01 20:02:00.163994 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.163962 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.166238 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.166206 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c04d0eff16222e7baef223e711863fea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"c04d0eff16222e7baef223e711863fea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.166306 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.166165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c04d0eff16222e7baef223e711863fea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"c04d0eff16222e7baef223e711863fea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.228656 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.228580 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:02:00.229625 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.229610 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.283826 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.283794 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:02:00.312002 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.311966 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.314616 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.314589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:02:00.696702 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.696616 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 20:02:00.697372 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.696860 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:02:00.697372 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.696862 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:02:00.697511 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:00.697125 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-78.us-west-1.compute.internal.18e404562dd9c430\": read tcp 10.0.128.78:45764->3.138.241.54:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-78.us-west-1.compute.internal.18e404562dd9c430 default 5320 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-78.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-78.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-78.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:01:59 +0000 UTC,LastTimestamp:2026-11-01 20:01:59.947926279 +0000 UTC m=+0.558107217,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-78.us-west-1.compute.internal,}" Nov 01 20:02:00.736288 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.736253 2573 apiserver.go:52] "Watching apiserver" Nov 01 20:02:00.763232 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.763202 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 20:02:00.980649 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.980556 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:57:00 +0000 UTC" deadline="2028-07-29 23:39:43.475260678 +0000 UTC" Nov 01 20:02:00.980649 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:00.980590 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15267h37m42.494674402s" Nov 01 20:02:01.000072 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.000050 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:02:01.000435 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.000414 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-f9xbg","kube-system/global-pull-secret-syncer-7lwmf","openshift-dns/node-resolver-ct9f5","openshift-image-registry/node-ca-zd4r8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-7bjpt","openshift-multus/network-metrics-daemon-qm6s6","openshift-network-diagnostics/network-check-target-ks9th","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv","openshift-multus/multus-2kxmx","kube-system/konnectivity-agent-8rbsz","kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-h8n5g","openshift-network-operator/iptables-alerter-2kzz5"] Nov 01 20:02:01.003052 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.003026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.005303 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.005280 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.005437 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.005361 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:01.007503 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.007473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.009547 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.009530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.011787 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.011768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.013864 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.013844 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:01.013955 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.013914 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:01.016035 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.016018 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:01.016117 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.016085 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:01.018223 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.018205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.020418 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.020399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.022657 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.022640 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.024918 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.024903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.027179 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.027162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.054910 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.054884 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:02:01.055089 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.055041 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\"" Nov 01 20:02:01.055429 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.055401 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:02:01.055614 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.055505 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:02:01.055614 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.055605 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:02:01.055614 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.055604 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:02:01.055823 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.055787 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:02:01.058585 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.058555 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:02:01.058760 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.058749 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:02:01.058823 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.058796 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\"" Nov 01 20:02:01.060424 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.060397 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:02:01.060517 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.060471 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\"" Nov 01 20:02:01.065721 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.065692 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:02:01.071301 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovn-node-metrics-cert\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.071432 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-lib-modules\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.071432 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071323 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-daemon-config\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.071432 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlt72\" (UniqueName: \"kubernetes.io/projected/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-kube-api-access-mlt72\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.071432 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-var-lib-kubelet\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.071555 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-kubelet\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.071555 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d979ea87-f591-49d1-a326-bb58c0af72aa-agent-certs\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.071555 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-systemd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.071640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071572 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.071640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgm5b\" (UniqueName: \"kubernetes.io/projected/4561dd90-478e-46b0-805e-b2ead5252a83-kube-api-access-hgm5b\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:01.071640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.071725 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.071725 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-ovn\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.071725 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-netns\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.071725 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-etc-kubernetes\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.071831 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5t6p\" (UniqueName: \"kubernetes.io/projected/0db527d3-b994-4c15-9a6e-3c01ea4b092e-kube-api-access-s5t6p\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.071831 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7pl\" (UniqueName: \"kubernetes.io/projected/5931bef0-3ad6-4758-82c5-352add308c77-kube-api-access-tg7pl\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.071831 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-modprobe-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.071913 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-multus-certs\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.071913 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db527d3-b994-4c15-9a6e-3c01ea4b092e-host-slash\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.071913 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-log-socket\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.071913 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-systemd-units\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.071913 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-netns\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072053 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072053 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:01.072053 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.071988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.072053 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-socket-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.072053 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-bin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.072224 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-kubelet\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072224 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.072224 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-systemd\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.072224 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-host\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.072224 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-slash\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072224 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-etc-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072224 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-dbus\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.072434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460d8bce-a4b3-4a43-b5cb-f7ff32651690-host\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.072434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.072434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.072434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d979ea87-f591-49d1-a326-bb58c0af72aa-konnectivity-ca\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.072434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-netd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-kubelet-config\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.072434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-device-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysconfig\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-hosts-file\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-tmp-dir\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072560 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-cnibin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-cni-binary-copy\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-node-log\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072638 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-registration-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-conf-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-bin\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.072714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhh8\" (UniqueName: \"kubernetes.io/projected/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-kube-api-access-rlhh8\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460d8bce-a4b3-4a43-b5cb-f7ff32651690-serviceca\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647zs\" (UniqueName: \"kubernetes.io/projected/460d8bce-a4b3-4a43-b5cb-f7ff32651690-kube-api-access-647zs\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-conf\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-system-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-var-lib-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-system-cni-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.072995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cnibin\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.073047 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073026 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-kubernetes\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv69v\" (UniqueName: \"kubernetes.io/projected/85aede5a-5dd5-4665-a7c8-f8efa9eea428-kube-api-access-rv69v\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-socket-dir-parent\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-multus\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-hostroot\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-config\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073260 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-env-overrides\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-script-lib\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-tuned\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.073430 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljcc7\" (UniqueName: \"kubernetes.io/projected/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-kube-api-access-ljcc7\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frr7p\" (UniqueName: \"kubernetes.io/projected/3a5a4a2b-455d-42ac-8064-cb987052b764-kube-api-access-frr7p\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-run\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-sys\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db527d3-b994-4c15-9a6e-3c01ea4b092e-iptables-alerter-script\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-sys-fs\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-tmp\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.073822 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.073727 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-k8s-cni-cncf-io\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.093965 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.093925 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:02:01.094087 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.094043 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:02:01.100577 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.100556 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:02:01.104469 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104436 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104599 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104618 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104621 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104643 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104646 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104660 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104622 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104684 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104694 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104773 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104781 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104815 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104824 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104815 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104849 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104869 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104882 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104888 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104903 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:02:01.104914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104926 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:02:01.105788 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.104967 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:02:01.105788 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.105017 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2p96j" Nov 01 20:02:01.174062 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-systemd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174102 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-systemd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgm5b\" (UniqueName: \"kubernetes.io/projected/4561dd90-478e-46b0-805e-b2ead5252a83-kube-api-access-hgm5b\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174153 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-ovn\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-netns\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.174239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-etc-kubernetes\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174251 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5t6p\" (UniqueName: \"kubernetes.io/projected/0db527d3-b994-4c15-9a6e-3c01ea4b092e-kube-api-access-s5t6p\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174306 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-netns\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174309 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-ovn\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174321 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-etc-kubernetes\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7pl\" (UniqueName: \"kubernetes.io/projected/5931bef0-3ad6-4758-82c5-352add308c77-kube-api-access-tg7pl\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174373 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-modprobe-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-multus-certs\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db527d3-b994-4c15-9a6e-3c01ea4b092e-host-slash\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-log-socket\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-systemd-units\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-netns\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174547 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db527d3-b994-4c15-9a6e-3c01ea4b092e-host-slash\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174552 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-log-socket\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174512 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-modprobe-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174580 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-netns\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.174640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-systemd-units\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174631 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174662 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-multus-certs\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-socket-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.174708 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-bin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174727 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174729 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-kubelet\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174776 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-bin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.174814 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs podName:4561dd90-478e-46b0-805e-b2ead5252a83 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:01.674760989 +0000 UTC m=+2.284941931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs") pod "network-metrics-daemon-qm6s6" (UID: "4561dd90-478e-46b0-805e-b2ead5252a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174802 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-kubelet\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-systemd\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.175466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174888 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-socket-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-host\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-slash\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174934 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-systemd\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-slash\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-etc-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.174981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-host\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-dbus\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175007 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-etc-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460d8bce-a4b3-4a43-b5cb-f7ff32651690-host\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175088 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460d8bce-a4b3-4a43-b5cb-f7ff32651690-host\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d979ea87-f591-49d1-a326-bb58c0af72aa-konnectivity-ca\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-netd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-kubelet-config\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175188 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-dbus\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-device-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.176239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175206 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysconfig\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175231 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-netd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-hosts-file\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175262 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-device-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-kubelet-config\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-tmp-dir\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175278 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysconfig\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-hosts-file\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-cnibin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-cni-binary-copy\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-node-log\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175399 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-cnibin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-registration-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175437 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-registration-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175442 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-node-log\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177066 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-conf-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-bin\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhh8\" (UniqueName: \"kubernetes.io/projected/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-kube-api-access-rlhh8\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175546 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-bin\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175549 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460d8bce-a4b3-4a43-b5cb-f7ff32651690-serviceca\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175554 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-conf-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175564 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-tmp-dir\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647zs\" (UniqueName: \"kubernetes.io/projected/460d8bce-a4b3-4a43-b5cb-f7ff32651690-kube-api-access-647zs\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-conf\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175618 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d979ea87-f591-49d1-a326-bb58c0af72aa-konnectivity-ca\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-system-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-var-lib-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-system-cni-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175717 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-conf\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.177751 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cnibin\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175786 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-system-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-system-cni-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-kubernetes\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175834 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-kubernetes\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv69v\" (UniqueName: \"kubernetes.io/projected/85aede5a-5dd5-4665-a7c8-f8efa9eea428-kube-api-access-rv69v\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175892 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cnibin\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-socket-dir-parent\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175927 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-socket-dir-parent\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175887 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-var-lib-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175930 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-multus\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175959 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-multus\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-hostroot\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.175999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-config\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-hostroot\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-env-overrides\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-script-lib\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.178219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-tuned\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcc7\" (UniqueName: \"kubernetes.io/projected/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-kube-api-access-ljcc7\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frr7p\" (UniqueName: \"kubernetes.io/projected/3a5a4a2b-455d-42ac-8064-cb987052b764-kube-api-access-frr7p\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-run\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-sys\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db527d3-b994-4c15-9a6e-3c01ea4b092e-iptables-alerter-script\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-sys-fs\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-tmp\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176356 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-run\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-k8s-cni-cncf-io\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovn-node-metrics-cert\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176423 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-sys\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-lib-modules\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176476 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176524 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-lib-modules\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176532 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-k8s-cni-cncf-io\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-daemon-config\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlt72\" (UniqueName: \"kubernetes.io/projected/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-kube-api-access-mlt72\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176599 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-sys-fs\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-var-lib-kubelet\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.176627 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-kubelet\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.176689 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret podName:7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee nodeName:}" failed. No retries permitted until 2026-11-01 20:02:01.676661413 +0000 UTC m=+2.286842339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret") pod "global-pull-secret-syncer-7lwmf" (UID: "7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176723 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-kubelet\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176882 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460d8bce-a4b3-4a43-b5cb-f7ff32651690-serviceca\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176971 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db527d3-b994-4c15-9a6e-3c01ea4b092e-iptables-alerter-script\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177109 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177165 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-script-lib\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177233 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-env-overrides\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177243 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-var-lib-kubelet\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.176722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d979ea87-f591-49d1-a326-bb58c0af72aa-agent-certs\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.179925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177386 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-config\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.180651 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177558 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-daemon-config\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.180651 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.177589 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-cni-binary-copy\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.180651 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.180279 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-tuned\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.180651 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.180350 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovn-node-metrics-cert\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.180651 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.180447 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d979ea87-f591-49d1-a326-bb58c0af72aa-agent-certs\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.180800 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.180714 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-tmp\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.230494 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.230264 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:01.230494 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.230296 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:01.230494 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.230312 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2rkgr for pod openshift-network-diagnostics/network-check-target-ks9th: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:01.230494 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.230404 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr podName:5f7737ff-1496-4eed-9bfb-b3e6ea260132 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:01.730382814 +0000 UTC m=+2.340563752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2rkgr" (UniqueName: "kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr") pod "network-check-target-ks9th" (UID: "5f7737ff-1496-4eed-9bfb-b3e6ea260132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:01.232235 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.232166 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5t6p\" (UniqueName: \"kubernetes.io/projected/0db527d3-b994-4c15-9a6e-3c01ea4b092e-kube-api-access-s5t6p\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.232392 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.232288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-647zs\" (UniqueName: \"kubernetes.io/projected/460d8bce-a4b3-4a43-b5cb-f7ff32651690-kube-api-access-647zs\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.232466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.232440 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7pl\" (UniqueName: \"kubernetes.io/projected/5931bef0-3ad6-4758-82c5-352add308c77-kube-api-access-tg7pl\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.232526 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.232475 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgm5b\" (UniqueName: \"kubernetes.io/projected/4561dd90-478e-46b0-805e-b2ead5252a83-kube-api-access-hgm5b\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:01.233412 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.233382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhh8\" (UniqueName: \"kubernetes.io/projected/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-kube-api-access-rlhh8\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.233506 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.233466 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv69v\" (UniqueName: \"kubernetes.io/projected/85aede5a-5dd5-4665-a7c8-f8efa9eea428-kube-api-access-rv69v\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.234127 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.234091 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlt72\" (UniqueName: \"kubernetes.io/projected/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-kube-api-access-mlt72\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.234454 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.234437 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcc7\" (UniqueName: \"kubernetes.io/projected/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-kube-api-access-ljcc7\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.234549 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.234533 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frr7p\" (UniqueName: \"kubernetes.io/projected/3a5a4a2b-455d-42ac-8064-cb987052b764-kube-api-access-frr7p\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.353031 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.352999 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:02:01.361132 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.361108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:01.371085 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.371057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:02:01.396537 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.396507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:02:01.398715 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.398684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:02:01.408905 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.408882 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kxmx" Nov 01 20:02:01.428303 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.428272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:01.432744 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.432662 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04d0eff16222e7baef223e711863fea.slice/crio-cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225 WatchSource:0}: Error finding container cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225: Status 404 returned error can't find the container with id cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225 Nov 01 20:02:01.432948 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.432923 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5931bef0_3ad6_4758_82c5_352add308c77.slice/crio-c7725c3b6cf4533801c588c89317e48d70b3a0850879a957c0794e5146418e84 WatchSource:0}: Error finding container c7725c3b6cf4533801c588c89317e48d70b3a0850879a957c0794e5146418e84: Status 404 returned error can't find the container with id c7725c3b6cf4533801c588c89317e48d70b3a0850879a957c0794e5146418e84 Nov 01 20:02:01.433312 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.433288 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:02:01.433469 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.433435 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460d8bce_a4b3_4a43_b5cb_f7ff32651690.slice/crio-9fedaa5297a99a3c3a39916c21582943e803408a4355d5943634acee92b277de WatchSource:0}: Error finding container 9fedaa5297a99a3c3a39916c21582943e803408a4355d5943634acee92b277de: Status 404 returned error can't find the container with id 9fedaa5297a99a3c3a39916c21582943e803408a4355d5943634acee92b277de Nov 01 20:02:01.434503 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.434483 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5a4a2b_455d_42ac_8064_cb987052b764.slice/crio-d40959f94916497c751f22e6b9995f21018fe0fda22912ebf7952e5c43ae4433 WatchSource:0}: Error finding container d40959f94916497c751f22e6b9995f21018fe0fda22912ebf7952e5c43ae4433: Status 404 returned error can't find the container with id d40959f94916497c751f22e6b9995f21018fe0fda22912ebf7952e5c43ae4433 Nov 01 20:02:01.435573 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.435529 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ad15ca_e56c_4ddf_9bac_a0243c4f85e5.slice/crio-db9c2d78f93b4e895f0749ac63cfecd5081b4a2d2387199c42117ca83b46853d WatchSource:0}: Error finding container db9c2d78f93b4e895f0749ac63cfecd5081b4a2d2387199c42117ca83b46853d: Status 404 returned error can't find the container with id db9c2d78f93b4e895f0749ac63cfecd5081b4a2d2387199c42117ca83b46853d Nov 01 20:02:01.436417 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.436375 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da5f29a924a82bf1fb8d408f5c8f1a7.slice/crio-94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd WatchSource:0}: Error finding container 94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd: Status 404 returned error can't find the container with id 94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd Nov 01 20:02:01.443462 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.443308 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:02:01.446898 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.446870 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd979ea87_f591_49d1_a326_bb58c0af72aa.slice/crio-3ac77269f540a824c3bb9ac53fe976f5a6ce8f4178f75492ff12ae62b9ba3d35 WatchSource:0}: Error finding container 3ac77269f540a824c3bb9ac53fe976f5a6ce8f4178f75492ff12ae62b9ba3d35: Status 404 returned error can't find the container with id 3ac77269f540a824c3bb9ac53fe976f5a6ce8f4178f75492ff12ae62b9ba3d35 Nov 01 20:02:01.447719 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.447694 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85aede5a_5dd5_4665_a7c8_f8efa9eea428.slice/crio-34221a57169d7276d126ce4dcc4337d26bf907f47b7512db92fef929d58aea7f WatchSource:0}: Error finding container 34221a57169d7276d126ce4dcc4337d26bf907f47b7512db92fef929d58aea7f: Status 404 returned error can't find the container with id 34221a57169d7276d126ce4dcc4337d26bf907f47b7512db92fef929d58aea7f Nov 01 20:02:01.455147 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.455124 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:02:01.465667 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:01.465642 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db527d3_b994_4c15_9a6e_3c01ea4b092e.slice/crio-a4c93e2c40fb25c207514420be7c29cb149d1b07952fdd6d50bbb2478a9b0831 WatchSource:0}: Error finding container a4c93e2c40fb25c207514420be7c29cb149d1b07952fdd6d50bbb2478a9b0831: Status 404 returned error can't find the container with id a4c93e2c40fb25c207514420be7c29cb149d1b07952fdd6d50bbb2478a9b0831 Nov 01 20:02:01.651792 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.651709 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:02:01.680136 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.680101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:01.680241 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.680148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:01.680295 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.680270 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:01.680295 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.680270 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:01.680403 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.680328 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs podName:4561dd90-478e-46b0-805e-b2ead5252a83 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:02.680313702 +0000 UTC m=+3.290494629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs") pod "network-metrics-daemon-qm6s6" (UID: "4561dd90-478e-46b0-805e-b2ead5252a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:01.680403 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.680342 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret podName:7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee nodeName:}" failed. No retries permitted until 2026-11-01 20:02:02.680335254 +0000 UTC m=+3.290516179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret") pod "global-pull-secret-syncer-7lwmf" (UID: "7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:01.698662 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.698621 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:02:01.781602 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.781565 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:01.781797 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.781708 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:01.781797 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.781723 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:01.781797 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.781733 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2rkgr for pod openshift-network-diagnostics/network-check-target-ks9th: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:01.781797 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:01.781785 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr podName:5f7737ff-1496-4eed-9bfb-b3e6ea260132 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:02.781772651 +0000 UTC m=+3.391953578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rkgr" (UniqueName: "kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr") pod "network-check-target-ks9th" (UID: "5f7737ff-1496-4eed-9bfb-b3e6ea260132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:01.865679 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.865622 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" event={"ID":"85aede5a-5dd5-4665-a7c8-f8efa9eea428","Type":"ContainerStarted","Data":"34221a57169d7276d126ce4dcc4337d26bf907f47b7512db92fef929d58aea7f"} Nov 01 20:02:01.866745 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.866716 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8rbsz" event={"ID":"d979ea87-f591-49d1-a326-bb58c0af72aa","Type":"ContainerStarted","Data":"3ac77269f540a824c3bb9ac53fe976f5a6ce8f4178f75492ff12ae62b9ba3d35"} Nov 01 20:02:01.868270 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.868246 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"9da5f29a924a82bf1fb8d408f5c8f1a7","Type":"ContainerStarted","Data":"94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd"} Nov 01 20:02:01.869404 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.869375 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kxmx" event={"ID":"3a5a4a2b-455d-42ac-8064-cb987052b764","Type":"ContainerStarted","Data":"d40959f94916497c751f22e6b9995f21018fe0fda22912ebf7952e5c43ae4433"} Nov 01 20:02:01.870477 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.870453 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerStarted","Data":"db9c2d78f93b4e895f0749ac63cfecd5081b4a2d2387199c42117ca83b46853d"} Nov 01 20:02:01.871514 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.871492 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"c04d0eff16222e7baef223e711863fea","Type":"ContainerStarted","Data":"cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225"} Nov 01 20:02:01.872599 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.872578 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kzz5" event={"ID":"0db527d3-b994-4c15-9a6e-3c01ea4b092e","Type":"ContainerStarted","Data":"a4c93e2c40fb25c207514420be7c29cb149d1b07952fdd6d50bbb2478a9b0831"} Nov 01 20:02:01.873721 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.873700 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" event={"ID":"5931bef0-3ad6-4758-82c5-352add308c77","Type":"ContainerStarted","Data":"c7725c3b6cf4533801c588c89317e48d70b3a0850879a957c0794e5146418e84"} Nov 01 20:02:01.874858 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:01.874831 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zd4r8" event={"ID":"460d8bce-a4b3-4a43-b5cb-f7ff32651690","Type":"ContainerStarted","Data":"9fedaa5297a99a3c3a39916c21582943e803408a4355d5943634acee92b277de"} Nov 01 20:02:02.108781 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.108642 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:57:01 +0000 UTC" deadline="2028-04-06 02:25:13.847485988 +0000 UTC" Nov 01 20:02:02.108781 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.108717 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12510h23m11.738772288s" Nov 01 20:02:02.267097 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:02.267053 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1bf24a_ab20_442b_b54a_b21dd4554ce2.slice/crio-5464920b79c38dbf826a9b3bc04dc33a62856daa653a58683209e26c50864fa5 WatchSource:0}: Error finding container 5464920b79c38dbf826a9b3bc04dc33a62856daa653a58683209e26c50864fa5: Status 404 returned error can't find the container with id 5464920b79c38dbf826a9b3bc04dc33a62856daa653a58683209e26c50864fa5 Nov 01 20:02:02.327402 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:02.327370 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac97caa_7b32_44f7_8ac6_01fe4a9db8ef.slice/crio-615cde627b6896c432dd15a2527790a8550808513d6fb140f101c7cae202ce09 WatchSource:0}: Error finding container 615cde627b6896c432dd15a2527790a8550808513d6fb140f101c7cae202ce09: Status 404 returned error can't find the container with id 615cde627b6896c432dd15a2527790a8550808513d6fb140f101c7cae202ce09 Nov 01 20:02:02.689176 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.689131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:02.689425 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.689257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:02.689502 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.689432 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:02.689565 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.689528 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret podName:7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee nodeName:}" failed. No retries permitted until 2026-11-01 20:02:04.689482733 +0000 UTC m=+5.299663676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret") pod "global-pull-secret-syncer-7lwmf" (UID: "7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:02.689633 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.689607 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:02.689688 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.689646 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs podName:4561dd90-478e-46b0-805e-b2ead5252a83 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:04.689633078 +0000 UTC m=+5.299814010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs") pod "network-metrics-daemon-qm6s6" (UID: "4561dd90-478e-46b0-805e-b2ead5252a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:02.790649 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.790609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:02.791259 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.790821 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:02.791259 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.790846 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:02.791259 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.790859 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2rkgr for pod openshift-network-diagnostics/network-check-target-ks9th: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:02.791259 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.790923 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr podName:5f7737ff-1496-4eed-9bfb-b3e6ea260132 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:04.79090352 +0000 UTC m=+5.401084450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rkgr" (UniqueName: "kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr") pod "network-check-target-ks9th" (UID: "5f7737ff-1496-4eed-9bfb-b3e6ea260132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:02.861496 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.861252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:02.861496 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.861259 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:02.861496 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.861453 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:02.861807 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.861581 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:02.861807 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.861425 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:02.861807 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:02.861714 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:02.882888 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.882824 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ct9f5" event={"ID":"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef","Type":"ContainerStarted","Data":"615cde627b6896c432dd15a2527790a8550808513d6fb140f101c7cae202ce09"} Nov 01 20:02:02.897105 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:02.897064 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"5464920b79c38dbf826a9b3bc04dc33a62856daa653a58683209e26c50864fa5"} Nov 01 20:02:03.109628 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:03.109530 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:57:01 +0000 UTC" deadline="2028-08-16 14:02:36.33053271 +0000 UTC" Nov 01 20:02:03.109628 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:03.109579 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15690h0m33.220959394s" Nov 01 20:02:04.708548 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:04.708502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:04.709127 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:04.708591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:04.709127 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.708769 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:04.709127 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.708844 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs podName:4561dd90-478e-46b0-805e-b2ead5252a83 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:08.708822523 +0000 UTC m=+9.319003452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs") pod "network-metrics-daemon-qm6s6" (UID: "4561dd90-478e-46b0-805e-b2ead5252a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:04.709127 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.708920 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:04.709127 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.708954 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret podName:7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee nodeName:}" failed. No retries permitted until 2026-11-01 20:02:08.708942367 +0000 UTC m=+9.319123292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret") pod "global-pull-secret-syncer-7lwmf" (UID: "7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:04.809861 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:04.809822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:04.810070 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.810040 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:04.810070 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.810070 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:04.810212 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.810084 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2rkgr for pod openshift-network-diagnostics/network-check-target-ks9th: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:04.810212 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.810149 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr podName:5f7737ff-1496-4eed-9bfb-b3e6ea260132 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:08.810129734 +0000 UTC m=+9.420310664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rkgr" (UniqueName: "kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr") pod "network-check-target-ks9th" (UID: "5f7737ff-1496-4eed-9bfb-b3e6ea260132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:04.864351 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:04.864314 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:04.864535 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:04.864314 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:04.864535 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.864490 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:04.864657 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:04.864536 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:04.865009 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.864650 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:04.865009 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:04.864906 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:05.925701 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:05.924612 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerStarted","Data":"7e770d18e2bbca5656dae7eb51ac3c40c6890abb51a6fd3911fdec28e858afed"} Nov 01 20:02:06.861790 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.861192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:06.861790 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:06.861414 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:06.861790 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.861487 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:06.861790 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:06.861568 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:06.861790 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.861617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:06.861790 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:06.861710 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:06.934155 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.934108 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8rbsz" event={"ID":"d979ea87-f591-49d1-a326-bb58c0af72aa","Type":"ContainerStarted","Data":"66cc20a7d8a4d16156f730f915072f436a212374eb60f0650e766865a385ec2f"} Nov 01 20:02:06.943647 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.943612 2573 generic.go:356] "Generic (PLEG): container finished" podID="9da5f29a924a82bf1fb8d408f5c8f1a7" containerID="2b3f4815ff02fce3d44fe5d03eb829d10338fc08e576b58fa927d8f53a34a84b" exitCode=0 Nov 01 20:02:06.943794 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.943713 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"9da5f29a924a82bf1fb8d408f5c8f1a7","Type":"ContainerDied","Data":"2b3f4815ff02fce3d44fe5d03eb829d10338fc08e576b58fa927d8f53a34a84b"} Nov 01 20:02:06.957640 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.957594 2573 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="7e770d18e2bbca5656dae7eb51ac3c40c6890abb51a6fd3911fdec28e858afed" exitCode=0 Nov 01 20:02:06.957847 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:06.957658 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerDied","Data":"7e770d18e2bbca5656dae7eb51ac3c40c6890abb51a6fd3911fdec28e858afed"} Nov 01 20:02:07.181502 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:07.181357 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8rbsz" podStartSLOduration=2.006018916 podStartE2EDuration="7.181334841s" totalImagesPullingTime="5.175315925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:07.0561882 +0000 UTC m=+7.666369150" watchObservedRunningTime="2026-11-01 20:02:07.181334841 +0000 UTC m=+7.791515793" Nov 01 20:02:07.964800 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:07.964759 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"c04d0eff16222e7baef223e711863fea","Type":"ContainerStarted","Data":"12481e4a0c1285da036c811dcbe36efa6ee4dfd7321c5ae15d10da7382e9d2ad"} Nov 01 20:02:08.205959 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.204807 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" podStartSLOduration=8.204784282 podStartE2EDuration="8.204784282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:08.085102764 +0000 UTC m=+8.695283715" watchObservedRunningTime="2026-11-01 20:02:08.204784282 +0000 UTC m=+8.814965234" Nov 01 20:02:08.745565 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.745525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:08.745764 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.745597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:08.745764 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.745699 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:08.745764 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.745724 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:08.745957 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.745786 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret podName:7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.745760829 +0000 UTC m=+17.355941775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret") pod "global-pull-secret-syncer-7lwmf" (UID: "7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:08.745957 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.745819 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs podName:4561dd90-478e-46b0-805e-b2ead5252a83 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.745798227 +0000 UTC m=+17.355979164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs") pod "network-metrics-daemon-qm6s6" (UID: "4561dd90-478e-46b0-805e-b2ead5252a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:08.847453 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.846932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:08.847453 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.847201 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:08.847453 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.847223 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:08.847453 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.847238 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2rkgr for pod openshift-network-diagnostics/network-check-target-ks9th: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:08.847453 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.847303 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr podName:5f7737ff-1496-4eed-9bfb-b3e6ea260132 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.847282849 +0000 UTC m=+17.457463783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rkgr" (UniqueName: "kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr") pod "network-check-target-ks9th" (UID: "5f7737ff-1496-4eed-9bfb-b3e6ea260132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:08.861984 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.861946 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:08.862167 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.862090 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:08.862245 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.862176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:08.862293 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.862262 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:08.862350 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.862331 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:08.862499 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:08.862449 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:08.970120 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.970068 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" event={"ID":"5931bef0-3ad6-4758-82c5-352add308c77","Type":"ContainerStarted","Data":"f783f33bf3a9965d6a2af1c2070655fb4458e392204436d3c5d34055c89ce326"} Nov 01 20:02:08.973935 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.973861 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zd4r8" event={"ID":"460d8bce-a4b3-4a43-b5cb-f7ff32651690","Type":"ContainerStarted","Data":"389d0820eefa2b0d53d27b2ae41b7e24e7832dc9ffdf19d40b305ce9f67a2d03"} Nov 01 20:02:08.977808 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:08.977771 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"9da5f29a924a82bf1fb8d408f5c8f1a7","Type":"ContainerStarted","Data":"276c62c3b4498e2d61ca2335e4f09ab67b341ff0cb18e93cced5f44f0ffff5e2"} Nov 01 20:02:09.220914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:09.220796 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zd4r8" podStartSLOduration=2.589658165 podStartE2EDuration="9.220774909s" totalImagesPullingTime="6.631116744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:09.0976986 +0000 UTC m=+9.707879549" watchObservedRunningTime="2026-11-01 20:02:09.220774909 +0000 UTC m=+9.830955862" Nov 01 20:02:09.290892 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:09.289654 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:09.291701 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:09.291422 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:09.414403 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:09.414330 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" podStartSLOduration=9.414307098 podStartE2EDuration="9.414307098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:09.222085731 +0000 UTC m=+9.832266682" watchObservedRunningTime="2026-11-01 20:02:09.414307098 +0000 UTC m=+10.024488070" Nov 01 20:02:10.861625 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:10.861598 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:10.862001 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:10.861718 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:10.862001 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:10.861790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:10.862001 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:10.861878 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:10.862001 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:10.861940 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:10.862315 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:10.862011 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:10.864097 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:10.864073 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:02:10.986317 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:10.986271 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kxmx" event={"ID":"3a5a4a2b-455d-42ac-8064-cb987052b764","Type":"ContainerStarted","Data":"4e0c1b391877f8b9c807314f5dc3c7eb4f906819a82655ec8b362e43b53838f0"} Nov 01 20:02:10.989841 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:10.989811 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" event={"ID":"5931bef0-3ad6-4758-82c5-352add308c77","Type":"ContainerStarted","Data":"86b6414e0afb9373ae4efbe0375af33bb7356e29b2220428ce4e844c6979b872"} Nov 01 20:02:11.099477 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:11.099421 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2kxmx" podStartSLOduration=2.044274776 podStartE2EDuration="11.099401397s" totalImagesPullingTime="9.055126621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:11.099094693 +0000 UTC m=+11.709275676" watchObservedRunningTime="2026-11-01 20:02:11.099401397 +0000 UTC m=+11.709582368" Nov 01 20:02:11.858587 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:11.857603 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:02:10.864095159Z","UUID":"7881826a-a762-482b-93e6-1ae18bf00147","Handler":null,"Name":"","Endpoint":""} Nov 01 20:02:11.859893 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:11.859871 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:02:11.860019 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:11.859903 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:02:11.998053 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:11.997956 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ct9f5" event={"ID":"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef","Type":"ContainerStarted","Data":"e2e6e64a440b4904319a667d325851cff3fa56e071c29c07ab76c995de96537a"} Nov 01 20:02:12.003975 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:12.003936 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerStarted","Data":"6980897864b690c2a7d7726e99493c1ec5177d36247fc5aceab84d126c53488a"} Nov 01 20:02:12.224888 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:12.224822 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ct9f5" podStartSLOduration=3.114152471 podStartE2EDuration="12.224801883s" totalImagesPullingTime="9.110649412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:12.109129194 +0000 UTC m=+12.719310141" watchObservedRunningTime="2026-11-01 20:02:12.224801883 +0000 UTC m=+12.834982830" Nov 01 20:02:12.862107 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:12.862065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:12.862295 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:12.862218 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:12.862394 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:12.862311 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:12.862461 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:12.862415 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:12.862523 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:12.862458 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:12.862583 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:12.862559 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:13.008012 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:13.007918 2573 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="6980897864b690c2a7d7726e99493c1ec5177d36247fc5aceab84d126c53488a" exitCode=0 Nov 01 20:02:13.008012 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:13.007995 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerDied","Data":"6980897864b690c2a7d7726e99493c1ec5177d36247fc5aceab84d126c53488a"} Nov 01 20:02:13.009641 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:13.009614 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kzz5" event={"ID":"0db527d3-b994-4c15-9a6e-3c01ea4b092e","Type":"ContainerStarted","Data":"bcd3bee70f52cc220a25254401183139c84ad88669cf40ca2def0530fb943b62"} Nov 01 20:02:13.017966 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:13.017935 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" event={"ID":"5931bef0-3ad6-4758-82c5-352add308c77","Type":"ContainerStarted","Data":"01b446af0c360345db1eebc085e775148a794b091a64e488a0352f24e73bc95a"} Nov 01 20:02:13.238101 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:13.237592 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" podStartSLOduration=2.691914284 podStartE2EDuration="13.2375733s" totalImagesPullingTime="10.545659016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:13.236058274 +0000 UTC m=+13.846239213" watchObservedRunningTime="2026-11-01 20:02:13.2375733 +0000 UTC m=+13.847754247" Nov 01 20:02:13.346669 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:13.346576 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2kzz5" podStartSLOduration=3.7496022079999998 podStartE2EDuration="13.346562119s" totalImagesPullingTime="9.596959911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:13.346483671 +0000 UTC m=+13.956664618" watchObservedRunningTime="2026-11-01 20:02:13.346562119 +0000 UTC m=+13.956743065" Nov 01 20:02:14.021067 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:14.021024 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" event={"ID":"85aede5a-5dd5-4665-a7c8-f8efa9eea428","Type":"ContainerStarted","Data":"fdf7f666afc0e1b3682d84e3b6885564ed845ee3f17df70771ed9191dd97d869"} Nov 01 20:02:14.134075 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:14.134018 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" podStartSLOduration=2.115294378 podStartE2EDuration="14.134003359s" totalImagesPullingTime="12.018708981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:14.133605811 +0000 UTC m=+14.743786759" watchObservedRunningTime="2026-11-01 20:02:14.134003359 +0000 UTC m=+14.744184307" Nov 01 20:02:14.861197 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:14.861113 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:14.861372 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:14.861214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:14.861372 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:14.861236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:14.861452 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:14.861372 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:14.861452 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:14.861431 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:14.861541 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:14.861504 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:15.026137 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:15.026081 2573 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="6b6c634515ec1cafe2144cd0e1d3d37327d98be3c3b5fc044526616854ab4f92" exitCode=0 Nov 01 20:02:15.026654 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:15.026161 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerDied","Data":"6b6c634515ec1cafe2144cd0e1d3d37327d98be3c3b5fc044526616854ab4f92"} Nov 01 20:02:16.815092 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:16.814833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:16.815560 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:16.815155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:16.815560 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.814996 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:16.815560 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.815258 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs podName:4561dd90-478e-46b0-805e-b2ead5252a83 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:32.815236266 +0000 UTC m=+33.425417192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs") pod "network-metrics-daemon-qm6s6" (UID: "4561dd90-478e-46b0-805e-b2ead5252a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:16.815560 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.815261 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:16.815560 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.815324 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret podName:7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee nodeName:}" failed. No retries permitted until 2026-11-01 20:02:32.815306361 +0000 UTC m=+33.425487302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret") pod "global-pull-secret-syncer-7lwmf" (UID: "7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:16.862145 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:16.862104 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:16.862350 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:16.862158 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:16.862350 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:16.862193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:16.862461 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.862326 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:16.862502 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.862460 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:16.862582 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.862564 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:16.916263 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:16.916219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:16.916483 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.916449 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:16.916483 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.916484 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:16.916679 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.916497 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2rkgr for pod openshift-network-diagnostics/network-check-target-ks9th: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:16.916679 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:16.916563 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr podName:5f7737ff-1496-4eed-9bfb-b3e6ea260132 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:32.916543793 +0000 UTC m=+33.526724724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rkgr" (UniqueName: "kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr") pod "network-check-target-ks9th" (UID: "5f7737ff-1496-4eed-9bfb-b3e6ea260132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:17.033546 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:17.033512 2573 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="1bce52648b308eb5910b7a3bc828d447dfbd34ea9b15c32a538e693366e9cfd5" exitCode=0 Nov 01 20:02:17.033713 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:17.033581 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerDied","Data":"1bce52648b308eb5910b7a3bc828d447dfbd34ea9b15c32a538e693366e9cfd5"} Nov 01 20:02:18.309947 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.309915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:18.500763 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.500556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:18.861777 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.861685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:18.861777 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.861685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:18.861993 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:18.861828 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:18.861993 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.861685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:18.861993 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:18.861889 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:18.861993 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:18.861953 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:18.925588 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.925552 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:18.925782 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.925707 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:18.926447 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:18.926426 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:02:19.048061 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:19.048028 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"57c2176f2fce7178de0734c2c59978832b181ed45382731d1e3a8c46f2848e78"} Nov 01 20:02:19.048237 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:19.048149 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"acc9b3332e6fe7fb7e6caffdcd9334ab7161b335d9bd174e418091eb08430cbb"} Nov 01 20:02:19.048237 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:19.048198 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"14feba4368ec4a2cd2c8d9eb66b4fb44a17b1a335a62b0b5d124561866d7bf11"} Nov 01 20:02:19.048363 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:19.048237 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"d399a3cf76a6539668d29c3d1802f276279f7c38b2f0b998a398f87428dcefff"} Nov 01 20:02:19.048363 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:19.048283 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"18fb0b58d2ede4b7f9a6827f96d4224c6e2160624660b4c4453b382d5514edad"} Nov 01 20:02:19.048363 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:19.048354 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"87bd941f7ef3261f6a9d3503086bd142bf699aa97bb4aaae0677fc7a983d729a"} Nov 01 20:02:19.099108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:19.099069 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:20.005769 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.005727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:20.052995 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.052954 2573 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="00a75bc40c67340fe2f340038461b1015c5369892f9a778238c9f313610a7904" exitCode=0 Nov 01 20:02:20.053128 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.053006 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerDied","Data":"00a75bc40c67340fe2f340038461b1015c5369892f9a778238c9f313610a7904"} Nov 01 20:02:20.138113 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.138077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:20.831987 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.831954 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:20.861277 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.861250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:20.861532 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.861324 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:20.861532 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:20.861441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:20.861532 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:20.861463 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:20.861532 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:20.861515 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:20.861684 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:20.861573 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:21.058589 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:21.058554 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"dab5d2f6e0010b36af2279aaefab555d58c7bae8367d4671bb38ad1669c32bc1"} Nov 01 20:02:21.061445 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:21.061418 2573 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="53e47a8b583acdd21eed7965d05e47dcaf5c830bce5e06f4832847093ae8d972" exitCode=0 Nov 01 20:02:21.061546 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:21.061454 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerDied","Data":"53e47a8b583acdd21eed7965d05e47dcaf5c830bce5e06f4832847093ae8d972"} Nov 01 20:02:22.066365 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:22.066307 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" event={"ID":"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5","Type":"ContainerStarted","Data":"6c643374ef525736a9b3330c9f111cb69aff5efce20925be02bc56579597caf3"} Nov 01 20:02:22.282827 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:22.282772 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7bjpt" podStartSLOduration=9.407296027 podStartE2EDuration="22.282757776s" totalImagesPullingTime="12.875461749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:22.282390434 +0000 UTC m=+22.892571379" watchObservedRunningTime="2026-11-01 20:02:22.282757776 +0000 UTC m=+22.892938722" Nov 01 20:02:22.862117 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:22.862083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:22.862305 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:22.862094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:22.862305 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:22.862217 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:22.862305 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:22.862233 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:22.862521 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:22.862307 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:22.862521 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:22.862376 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:23.068263 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.068007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:23.071912 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.071887 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" event={"ID":"9b1bf24a-ab20-442b-b54a-b21dd4554ce2","Type":"ContainerStarted","Data":"69dcea750aede334d1cbd4b71ababea06c6107d82c37d2d4a9c778e30626b492"} Nov 01 20:02:23.213030 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.212997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:23.340642 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.340541 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" podStartSLOduration=7.329316921 podStartE2EDuration="23.340527081s" totalImagesPullingTime="16.01121016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:23.340295432 +0000 UTC m=+23.950476379" watchObservedRunningTime="2026-11-01 20:02:23.340527081 +0000 UTC m=+23.950708054" Nov 01 20:02:23.414042 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.414008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:23.537502 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.537469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:23.738050 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.738020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:23.865924 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:23.865889 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:24.009055 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.008967 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:24.074629 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.074600 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:24.075118 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.075091 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:24.075262 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.075168 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:24.092547 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.092514 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:24.092707 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.092627 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:24.135625 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.135597 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:24.259873 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.259775 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:24.862056 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.862024 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:24.862222 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.862097 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:24.862222 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:24.862123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:24.862222 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:24.862206 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:24.862341 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:24.862241 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:24.862341 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:24.862293 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:25.076535 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.076510 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:25.440266 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.440194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:25.570567 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.570534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:25.594967 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.594927 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7lwmf"] Nov 01 20:02:25.595123 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.595042 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:25.595172 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:25.595142 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:25.595857 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.595832 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ks9th"] Nov 01 20:02:25.595969 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.595925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:25.596041 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:25.596022 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:25.596414 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.596400 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qm6s6"] Nov 01 20:02:25.596479 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.596470 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:25.596555 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:25.596540 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:25.982969 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:25.982932 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:26.078811 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:26.078786 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:26.861674 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:26.861627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:26.861674 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:26.861679 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:26.861931 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:26.861763 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:26.861931 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:26.861898 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:27.033184 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:27.033148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:27.861283 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:27.861238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:27.861780 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:27.861371 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:28.861123 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:28.861082 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:28.861326 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:28.861082 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:28.861326 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:28.861220 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:28.861326 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:28.861265 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:29.095037 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:29.095007 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ct9f5_3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef/dns-node-resolver/0.log" Nov 01 20:02:29.863113 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:29.863079 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:29.863500 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:29.863202 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:30.036414 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:30.036381 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:30.036602 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:30.036579 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:30.048748 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:30.048726 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:02:30.095148 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:30.095118 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zd4r8_460d8bce-a4b3-4a43-b5cb-f7ff32651690/node-ca/0.log" Nov 01 20:02:30.861997 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:30.861827 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:30.862142 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:30.861881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:30.862142 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:30.862077 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:30.862237 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:30.862145 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:31.862093 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:31.862062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:31.862521 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:31.862172 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:32.827672 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:32.827638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:32.827872 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:32.827691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:32.827872 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.827800 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:32.827872 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.827854 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret podName:7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee nodeName:}" failed. No retries permitted until 2026-11-01 20:03:04.82784155 +0000 UTC m=+65.438022474 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret") pod "global-pull-secret-syncer-7lwmf" (UID: "7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:32.828007 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.827800 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:32.828007 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.827915 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs podName:4561dd90-478e-46b0-805e-b2ead5252a83 nodeName:}" failed. No retries permitted until 2026-11-01 20:03:04.827904355 +0000 UTC m=+65.438085292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs") pod "network-metrics-daemon-qm6s6" (UID: "4561dd90-478e-46b0-805e-b2ead5252a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:32.861954 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:32.861919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:32.861954 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:32.861949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:32.862148 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.862050 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:32.862148 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.862127 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:32.928347 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:32.928292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:32.928540 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.928506 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:32.928540 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.928531 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:32.928660 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.928545 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2rkgr for pod openshift-network-diagnostics/network-check-target-ks9th: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:32.928660 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:32.928611 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr podName:5f7737ff-1496-4eed-9bfb-b3e6ea260132 nodeName:}" failed. No retries permitted until 2026-11-01 20:03:04.928592271 +0000 UTC m=+65.538773209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rkgr" (UniqueName: "kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr") pod "network-check-target-ks9th" (UID: "5f7737ff-1496-4eed-9bfb-b3e6ea260132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:33.861801 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:33.861764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:33.861971 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:33.861903 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:34.862086 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:34.862051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:34.862493 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:34.862168 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:34.862493 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:34.862197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:34.862493 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:34.862256 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:35.861353 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:35.861302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:35.861512 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:35.861431 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:36.861925 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:36.861887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:36.862351 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:36.862007 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:36.862351 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:36.862037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:36.862351 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:36.862117 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:37.704229 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:37.704198 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:37.862046 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:37.862011 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:37.862445 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:37.862119 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:37.868687 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:37.868663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:37.998233 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:37.998187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:38.194365 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:38.194315 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:38.862060 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:38.862031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:38.862468 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:38.862181 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:38.862468 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:38.862221 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:38.862468 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:38.862285 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:39.862086 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:39.862051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:39.862594 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:39.862155 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:40.126235 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.126150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:40.254262 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.254220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:40.380860 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.380771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:40.508558 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.508523 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:40.633571 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.633490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:40.861833 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.861802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:40.862020 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:40.861928 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:40.862020 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.861970 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:40.862134 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:40.862064 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:40.880321 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:40.878254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:41.002313 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:41.002278 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:41.312607 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:41.312522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:41.436399 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:41.436365 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:41.660583 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:41.660508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:41.805661 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:41.805625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:41.861891 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:41.861852 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:41.862038 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:41.861992 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:42.862033 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:42.861990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:42.862033 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:42.862015 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:42.862484 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:42.862126 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:42.862484 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:42.862203 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:42.961806 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:42.961771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:43.085130 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:43.085092 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:43.861934 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:43.861897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:43.862129 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:43.862027 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:44.861394 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:44.861362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:44.861578 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:44.861401 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:44.861578 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:44.861468 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ks9th" podUID="5f7737ff-1496-4eed-9bfb-b3e6ea260132" Nov 01 20:02:44.861675 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:44.861634 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qm6s6" podUID="4561dd90-478e-46b0-805e-b2ead5252a83" Nov 01 20:02:45.071621 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:45.071586 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:45.207121 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:45.207031 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:45.591562 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:45.591526 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:45.717407 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:45.717376 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:45.861602 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:45.861511 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:45.861763 ip-10-0-128-78 kubenswrapper[2573]: E1101 20:02:45.861645 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lwmf" podUID="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" Nov 01 20:02:46.676099 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.676001 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeReady" Nov 01 20:02:46.676599 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.676157 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 20:02:46.830810 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.830779 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2v4wp"] Nov 01 20:02:46.832672 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.832648 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jwwcz"] Nov 01 20:02:46.832805 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.832765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:46.834671 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.834555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:46.837106 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.837085 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lthd5"] Nov 01 20:02:46.838487 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.838470 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2v4wp"] Nov 01 20:02:46.838629 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.838503 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jwwcz"] Nov 01 20:02:46.838629 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.838598 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:46.861601 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.861572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:02:46.861705 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.861610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:02:46.885166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885132 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:02:46.885296 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885149 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\"" Nov 01 20:02:46.885296 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885151 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:02:46.885296 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885153 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:02:46.885921 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885904 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:02:46.885921 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885916 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:02:46.886038 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885917 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:02:46.886038 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.885961 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:02:46.886774 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.886758 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\"" Nov 01 20:02:46.891544 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.891525 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lthd5"] Nov 01 20:02:46.928082 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.927993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93e86161-3b78-4cd6-960d-70755fcc7b76-metrics-tls\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:46.928082 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e86d2ca4-1f79-454b-bd82-e86db21a1abc-cert\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:46.928082 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmnv\" (UniqueName: \"kubernetes.io/projected/e86d2ca4-1f79-454b-bd82-e86db21a1abc-kube-api-access-4wmnv\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:46.928082 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4zh\" (UniqueName: \"kubernetes.io/projected/93e86161-3b78-4cd6-960d-70755fcc7b76-kube-api-access-jm4zh\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:46.928425 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtwv\" (UniqueName: \"kubernetes.io/projected/f0851759-372d-4706-8bf7-ee6a493872f4-kube-api-access-rqtwv\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:46.928425 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e86161-3b78-4cd6-960d-70755fcc7b76-tmp-dir\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:46.928425 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928299 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93e86161-3b78-4cd6-960d-70755fcc7b76-config-volume\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:46.928425 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0851759-372d-4706-8bf7-ee6a493872f4-data-volume\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:46.928554 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0851759-372d-4706-8bf7-ee6a493872f4-crio-socket\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:46.928554 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0851759-372d-4706-8bf7-ee6a493872f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:46.928554 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.928463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0851759-372d-4706-8bf7-ee6a493872f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:46.986181 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986130 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:02:46.986382 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986203 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" Nov 01 20:02:46.986382 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986231 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:02:46.986382 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986250 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:02:46.986382 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986257 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:02:46.986382 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986381 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\"" Nov 01 20:02:46.986579 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986437 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" Nov 01 20:02:46.986579 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:46.986507 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:02:47.028773 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.028739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e86d2ca4-1f79-454b-bd82-e86db21a1abc-cert\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:47.028773 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.028773 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmnv\" (UniqueName: \"kubernetes.io/projected/e86d2ca4-1f79-454b-bd82-e86db21a1abc-kube-api-access-4wmnv\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:47.028956 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.028801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4zh\" (UniqueName: \"kubernetes.io/projected/93e86161-3b78-4cd6-960d-70755fcc7b76-kube-api-access-jm4zh\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.028956 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.028846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtwv\" (UniqueName: \"kubernetes.io/projected/f0851759-372d-4706-8bf7-ee6a493872f4-kube-api-access-rqtwv\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.028956 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.028872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e86161-3b78-4cd6-960d-70755fcc7b76-tmp-dir\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.028956 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.028919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93e86161-3b78-4cd6-960d-70755fcc7b76-config-volume\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.029319 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029295 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e86161-3b78-4cd6-960d-70755fcc7b76-tmp-dir\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.029447 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029359 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0851759-372d-4706-8bf7-ee6a493872f4-data-volume\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.029447 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.028940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0851759-372d-4706-8bf7-ee6a493872f4-data-volume\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.029543 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0851759-372d-4706-8bf7-ee6a493872f4-crio-socket\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.029543 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029499 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0851759-372d-4706-8bf7-ee6a493872f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.029543 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0851759-372d-4706-8bf7-ee6a493872f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.029653 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93e86161-3b78-4cd6-960d-70755fcc7b76-metrics-tls\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.029697 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029648 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0851759-372d-4706-8bf7-ee6a493872f4-crio-socket\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.029697 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029673 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93e86161-3b78-4cd6-960d-70755fcc7b76-config-volume\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.029914 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.029897 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0851759-372d-4706-8bf7-ee6a493872f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.032878 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.032856 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93e86161-3b78-4cd6-960d-70755fcc7b76-metrics-tls\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.032974 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.032926 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0851759-372d-4706-8bf7-ee6a493872f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.033113 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.033096 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e86d2ca4-1f79-454b-bd82-e86db21a1abc-cert\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:47.085634 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.085601 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4zh\" (UniqueName: \"kubernetes.io/projected/93e86161-3b78-4cd6-960d-70755fcc7b76-kube-api-access-jm4zh\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.085818 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.085790 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtwv\" (UniqueName: \"kubernetes.io/projected/f0851759-372d-4706-8bf7-ee6a493872f4-kube-api-access-rqtwv\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.085944 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.085928 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmnv\" (UniqueName: \"kubernetes.io/projected/e86d2ca4-1f79-454b-bd82-e86db21a1abc-kube-api-access-4wmnv\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:47.145352 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.145306 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:02:47.152747 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.152725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:02:47.156642 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.156619 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:47.528673 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.528639 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2v4wp"] Nov 01 20:02:47.532311 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:47.532267 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86d2ca4_1f79_454b_bd82_e86db21a1abc.slice/crio-f0d3a8b55cb40dfd470791821223c8cbbd9f5f27ffc5ab3e2feb8d5aa80e8396 WatchSource:0}: Error finding container f0d3a8b55cb40dfd470791821223c8cbbd9f5f27ffc5ab3e2feb8d5aa80e8396: Status 404 returned error can't find the container with id f0d3a8b55cb40dfd470791821223c8cbbd9f5f27ffc5ab3e2feb8d5aa80e8396 Nov 01 20:02:47.580360 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.580303 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lthd5"] Nov 01 20:02:47.582854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.582833 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jwwcz"] Nov 01 20:02:47.584674 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:47.584569 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e86161_3b78_4cd6_960d_70755fcc7b76.slice/crio-67304fd16b169c449dc4c31ef68631157af0ed6d0956b91f961f0ca09993cc26 WatchSource:0}: Error finding container 67304fd16b169c449dc4c31ef68631157af0ed6d0956b91f961f0ca09993cc26: Status 404 returned error can't find the container with id 67304fd16b169c449dc4c31ef68631157af0ed6d0956b91f961f0ca09993cc26 Nov 01 20:02:47.586773 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:47.586747 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0851759_372d_4706_8bf7_ee6a493872f4.slice/crio-8658d7e408b8697cf601f810e2d9ca3c27e1f0eb5e2d88020910a2d9b6f8fe69 WatchSource:0}: Error finding container 8658d7e408b8697cf601f810e2d9ca3c27e1f0eb5e2d88020910a2d9b6f8fe69: Status 404 returned error can't find the container with id 8658d7e408b8697cf601f810e2d9ca3c27e1f0eb5e2d88020910a2d9b6f8fe69 Nov 01 20:02:47.757591 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.757554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:47.861777 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.861686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:02:47.915623 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.915591 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:02:47.915784 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:47.915657 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" Nov 01 20:02:48.130876 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:48.130791 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lthd5" event={"ID":"93e86161-3b78-4cd6-960d-70755fcc7b76","Type":"ContainerStarted","Data":"67304fd16b169c449dc4c31ef68631157af0ed6d0956b91f961f0ca09993cc26"} Nov 01 20:02:48.132261 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:48.132234 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwwcz" event={"ID":"f0851759-372d-4706-8bf7-ee6a493872f4","Type":"ContainerStarted","Data":"0fc25c70b4e9ed6ceb9a1ade6c651b2db5cd16fc4349196e575e246146b7e0e7"} Nov 01 20:02:48.132412 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:48.132289 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwwcz" event={"ID":"f0851759-372d-4706-8bf7-ee6a493872f4","Type":"ContainerStarted","Data":"8658d7e408b8697cf601f810e2d9ca3c27e1f0eb5e2d88020910a2d9b6f8fe69"} Nov 01 20:02:48.133401 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:48.133375 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2v4wp" event={"ID":"e86d2ca4-1f79-454b-bd82-e86db21a1abc","Type":"ContainerStarted","Data":"f0d3a8b55cb40dfd470791821223c8cbbd9f5f27ffc5ab3e2feb8d5aa80e8396"} Nov 01 20:02:49.138560 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:49.138507 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwwcz" event={"ID":"f0851759-372d-4706-8bf7-ee6a493872f4","Type":"ContainerStarted","Data":"d1155ffe7abac9ccd7e7d68b777772d18e3445ff219c9a0fc3367f17849bda45"} Nov 01 20:02:49.957261 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:49.957196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:50.091407 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.091378 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:50.142962 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.142927 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lthd5" event={"ID":"93e86161-3b78-4cd6-960d-70755fcc7b76","Type":"ContainerStarted","Data":"95bd31cf17781c16c5478835738c47bbb0e050af7592cf85e65b1b1984c62303"} Nov 01 20:02:50.143355 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.142990 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lthd5" event={"ID":"93e86161-3b78-4cd6-960d-70755fcc7b76","Type":"ContainerStarted","Data":"e565eef55aab7e6880bb07e8972a90ad3865bd2e6c116faab6d14b8fd83404ee"} Nov 01 20:02:50.143501 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.143474 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lthd5" Nov 01 20:02:50.213977 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.213917 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:50.378783 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.378748 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv"] Nov 01 20:02:50.380949 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.380924 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.405219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.405186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:50.433239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.433203 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 20:02:50.433239 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.433215 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jh6hf\"" Nov 01 20:02:50.433516 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.433363 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 20:02:50.433516 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.433417 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 20:02:50.433516 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.433462 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 20:02:50.433516 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.433476 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 20:02:50.438799 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.438767 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv"] Nov 01 20:02:50.555522 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.555429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlgj\" (UniqueName: \"kubernetes.io/projected/8b05d35e-918b-4b8c-9643-78f8dff0b109-kube-api-access-cjlgj\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.555684 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.555571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.555684 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.555618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.555684 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.555653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b05d35e-918b-4b8c-9643-78f8dff0b109-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.656614 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.656569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlgj\" (UniqueName: \"kubernetes.io/projected/8b05d35e-918b-4b8c-9643-78f8dff0b109-kube-api-access-cjlgj\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.656797 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.656640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.656797 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.656666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.656797 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.656692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b05d35e-918b-4b8c-9643-78f8dff0b109-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.657427 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.657404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b05d35e-918b-4b8c-9643-78f8dff0b109-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.660529 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.660506 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.661831 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.661799 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.713885 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.713863 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlgj\" (UniqueName: \"kubernetes.io/projected/8b05d35e-918b-4b8c-9643-78f8dff0b109-kube-api-access-cjlgj\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:50.996872 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:50.996832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:02:51.124674 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:51.124647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:51.147702 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:51.147620 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwwcz" event={"ID":"f0851759-372d-4706-8bf7-ee6a493872f4","Type":"ContainerStarted","Data":"6c028b721ac1a6a55f9634211b67669e48bf00071721d56aff8d13cc8ecd4092"} Nov 01 20:02:51.149306 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:51.149271 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2v4wp" event={"ID":"e86d2ca4-1f79-454b-bd82-e86db21a1abc","Type":"ContainerStarted","Data":"9f9e9d762002399c6d68172b849594c622b09fac78a5a84965e1655dd4f684d6"} Nov 01 20:02:51.404145 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:51.404031 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2v4wp" podStartSLOduration=2.056363734 podStartE2EDuration="5.404017022s" totalImagesPullingTime="3.347653288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:51.292018703 +0000 UTC m=+51.902199649" watchObservedRunningTime="2026-11-01 20:02:51.404017022 +0000 UTC m=+52.014197965" Nov 01 20:02:51.518189 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:51.518139 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jwwcz" podStartSLOduration=2.546339813 podStartE2EDuration="5.518124376s" totalImagesPullingTime="2.971784563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:51.51701694 +0000 UTC m=+52.127197886" watchObservedRunningTime="2026-11-01 20:02:51.518124376 +0000 UTC m=+52.128305322" Nov 01 20:02:51.538298 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:51.538268 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv"] Nov 01 20:02:51.541215 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:51.541176 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b05d35e_918b_4b8c_9643_78f8dff0b109.slice/crio-dc2fb59201257eefec5a5fbbe040c89fc1e0cb7eaad7ad6480f8b3b1395aacb7 WatchSource:0}: Error finding container dc2fb59201257eefec5a5fbbe040c89fc1e0cb7eaad7ad6480f8b3b1395aacb7: Status 404 returned error can't find the container with id dc2fb59201257eefec5a5fbbe040c89fc1e0cb7eaad7ad6480f8b3b1395aacb7 Nov 01 20:02:51.631915 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:51.631858 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lthd5" podStartSLOduration=3.320799178 podStartE2EDuration="5.631841138s" totalImagesPullingTime="2.31104196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:51.631088535 +0000 UTC m=+52.241269482" watchObservedRunningTime="2026-11-01 20:02:51.631841138 +0000 UTC m=+52.242022078" Nov 01 20:02:52.152315 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:52.152282 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" event={"ID":"8b05d35e-918b-4b8c-9643-78f8dff0b109","Type":"ContainerStarted","Data":"dc2fb59201257eefec5a5fbbe040c89fc1e0cb7eaad7ad6480f8b3b1395aacb7"} Nov 01 20:02:54.159864 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:54.159821 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" event={"ID":"8b05d35e-918b-4b8c-9643-78f8dff0b109","Type":"ContainerStarted","Data":"963f6ef92777bec5abdcb03df0793ceb81f1930001362c3e0ba48685d5b0329e"} Nov 01 20:02:54.160257 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:54.159886 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" event={"ID":"8b05d35e-918b-4b8c-9643-78f8dff0b109","Type":"ContainerStarted","Data":"271c5f015edbcae0a314793d8b918e67c1907ec3c1a958d7686eb5e42d8c6cba"} Nov 01 20:02:54.272307 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:54.272253 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" podStartSLOduration=2.38683969 podStartE2EDuration="4.272240791s" totalImagesPullingTime="1.885401101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:54.272147055 +0000 UTC m=+54.882328026" watchObservedRunningTime="2026-11-01 20:02:54.272240791 +0000 UTC m=+54.882421737" Nov 01 20:02:55.473616 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:55.473580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:55.630067 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:55.630032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:56.038646 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.038605 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:56.301392 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.301286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:56.537958 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.537928 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:56.656294 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.656209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:56.780602 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.780568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:56.906791 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.906712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:56.976249 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.976213 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9hnrf"] Nov 01 20:02:56.978280 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:56.978262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.025617 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.025586 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr"] Nov 01 20:02:57.027535 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.027519 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.039033 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.039011 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 20:02:57.074558 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.074532 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 20:02:57.074558 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.074535 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 20:02:57.074710 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.074612 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\"" Nov 01 20:02:57.081903 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.081878 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr"] Nov 01 20:02:57.100396 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.100325 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:57.105434 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-root\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105550 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-sys\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105550 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-textfile\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105617 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-tls\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105617 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-wtmp\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105679 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105679 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105869 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np598\" (UniqueName: \"kubernetes.io/projected/4931755e-e9dc-4a2f-aab7-5d64dacb361e-kube-api-access-np598\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.105899 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.105880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-metrics-client-ca\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.124131 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.124099 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 20:02:57.124131 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.124105 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 20:02:57.124293 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.124189 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-76lth\"" Nov 01 20:02:57.132600 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.132577 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 20:02:57.206356 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.206356 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206297 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-metrics-client-ca\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206356 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-textfile\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206638 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206638 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shpz\" (UniqueName: \"kubernetes.io/projected/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-api-access-7shpz\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.206638 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-sys\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206638 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206514 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-sys\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206638 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.206851 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np598\" (UniqueName: \"kubernetes.io/projected/4931755e-e9dc-4a2f-aab7-5d64dacb361e-kube-api-access-np598\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206851 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-root\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206851 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-tls\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206851 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206775 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-root\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.206851 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.206851 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206806 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-textfile\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.207097 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206938 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.207097 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.207097 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.206990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-wtmp\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.207097 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.207005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.207097 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.207013 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-metrics-client-ca\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.207097 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.207031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2c85daa-8760-4950-8cb2-e07c2aa5292e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.207281 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.207205 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-wtmp\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.209457 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.209408 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.209623 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.209602 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-tls\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.238784 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.238748 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:57.262677 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.262655 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np598\" (UniqueName: \"kubernetes.io/projected/4931755e-e9dc-4a2f-aab7-5d64dacb361e-kube-api-access-np598\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.289143 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.289101 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:02:57.299345 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:57.299299 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4931755e_e9dc_4a2f_aab7_5d64dacb361e.slice/crio-b2a333f0c31ea725139b310d3b8ca0a0833f522d72ee683aa320c99d9b213699 WatchSource:0}: Error finding container b2a333f0c31ea725139b310d3b8ca0a0833f522d72ee683aa320c99d9b213699: Status 404 returned error can't find the container with id b2a333f0c31ea725139b310d3b8ca0a0833f522d72ee683aa320c99d9b213699 Nov 01 20:02:57.307427 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.307401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shpz\" (UniqueName: \"kubernetes.io/projected/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-api-access-7shpz\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.307539 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.307445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.307539 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.307493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.307539 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.307511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.307539 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.307536 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2c85daa-8760-4950-8cb2-e07c2aa5292e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.307731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.307563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.308087 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.308035 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2c85daa-8760-4950-8cb2-e07c2aa5292e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.308251 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.308229 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.308369 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.308313 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.310060 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.310042 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.310122 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.310082 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.363445 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.363407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shpz\" (UniqueName: \"kubernetes.io/projected/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-api-access-7shpz\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.367021 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.366991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:57.494811 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.494771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:57.614638 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.614602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:57.638551 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.638513 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:02:57.736864 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.736824 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:57.954807 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:57.954728 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:58.055955 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:58.055927 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr"] Nov 01 20:02:58.060620 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:02:58.060592 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c85daa_8760_4950_8cb2_e07c2aa5292e.slice/crio-8c049b5fae3b06e7b163dcb50beff63b937223e0c301b4b9421c7b8664d65d7b WatchSource:0}: Error finding container 8c049b5fae3b06e7b163dcb50beff63b937223e0c301b4b9421c7b8664d65d7b: Status 404 returned error can't find the container with id 8c049b5fae3b06e7b163dcb50beff63b937223e0c301b4b9421c7b8664d65d7b Nov 01 20:02:58.080064 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:58.080036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:58.177446 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:58.177407 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hnrf" event={"ID":"4931755e-e9dc-4a2f-aab7-5d64dacb361e","Type":"ContainerStarted","Data":"b2a333f0c31ea725139b310d3b8ca0a0833f522d72ee683aa320c99d9b213699"} Nov 01 20:02:58.178466 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:58.178445 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" event={"ID":"f2c85daa-8760-4950-8cb2-e07c2aa5292e","Type":"ContainerStarted","Data":"8c049b5fae3b06e7b163dcb50beff63b937223e0c301b4b9421c7b8664d65d7b"} Nov 01 20:02:58.441808 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:58.441780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:58.675200 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:58.675169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:59.176321 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.176282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:59.327038 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.326998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:59.408676 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.408639 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:02:59.410929 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.410906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.422942 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.422917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423069 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.422948 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfggb\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423069 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.422976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423139 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423171 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423171 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423226 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423226 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423348 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423382 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423416 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423416 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.423474 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.423438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.464057 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.463997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:02:59.466007 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.465985 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 20:02:59.466222 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.466090 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 20:02:59.466222 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.466144 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 20:02:59.466222 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.466188 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 20:02:59.515608 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.515573 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 20:02:59.515608 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.515575 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ljq7\"" Nov 01 20:02:59.515931 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.515915 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 20:02:59.515994 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.515929 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 20:02:59.515994 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.515930 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 20:02:59.523907 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.523879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.523907 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.523910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.523930 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.523951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.523978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524108 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524458 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524458 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfggb\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524458 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.524458 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.524225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.525314 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.525284 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.526013 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.525982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.528352 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.528309 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.528642 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.528387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.528642 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.528447 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.528865 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.528691 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.529069 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.529034 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.529263 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.529245 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.529714 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.529690 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.529791 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.529762 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.530049 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.530027 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.566012 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.565974 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 20:02:59.566798 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.566774 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:02:59.575219 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.575195 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.580635 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.580606 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfggb\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:02:59.726015 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:02:59.725935 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:00.159851 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:00.159826 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:03:00.162549 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:03:00.162519 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1f68dc_829d_48d3_bfe8_71e7ca8f5b49.slice/crio-6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4 WatchSource:0}: Error finding container 6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4: Status 404 returned error can't find the container with id 6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4 Nov 01 20:03:00.186951 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:00.186919 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerStarted","Data":"6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4"} Nov 01 20:03:00.189398 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:00.189371 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" event={"ID":"f2c85daa-8760-4950-8cb2-e07c2aa5292e","Type":"ContainerStarted","Data":"4c6c3295eabfd5d5abc9e871ccbeca3cb175e8a2957293f8c55a4feaf9bac2d9"} Nov 01 20:03:00.189512 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:00.189428 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" event={"ID":"f2c85daa-8760-4950-8cb2-e07c2aa5292e","Type":"ContainerStarted","Data":"b5b5482ebfe62ee5a58635a42938a9e11139d5c3e5ef962ede5a1411c35b561f"} Nov 01 20:03:00.189512 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:00.189458 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" event={"ID":"f2c85daa-8760-4950-8cb2-e07c2aa5292e","Type":"ContainerStarted","Data":"765650003f88e6cdeb456f0265decb0537ea765747b7b14f43376ea6900af703"} Nov 01 20:03:00.191245 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:00.191184 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hnrf" event={"ID":"4931755e-e9dc-4a2f-aab7-5d64dacb361e","Type":"ContainerStarted","Data":"d0d924d27e1d42f3b666e76386101cf4afa1a8d207c244b9a4abc647bc099fff"} Nov 01 20:03:00.303555 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:00.303512 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" podStartSLOduration=2.448956228 podStartE2EDuration="4.303499071s" totalImagesPullingTime="1.854542843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:00.302482295 +0000 UTC m=+60.912663239" watchObservedRunningTime="2026-11-01 20:03:00.303499071 +0000 UTC m=+60.913680017" Nov 01 20:03:01.097708 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.097669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:01.154996 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.154964 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:01.197138 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.197049 2573 generic.go:356] "Generic (PLEG): container finished" podID="4931755e-e9dc-4a2f-aab7-5d64dacb361e" containerID="d0d924d27e1d42f3b666e76386101cf4afa1a8d207c244b9a4abc647bc099fff" exitCode=0 Nov 01 20:03:01.197314 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.197141 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hnrf" event={"ID":"4931755e-e9dc-4a2f-aab7-5d64dacb361e","Type":"ContainerDied","Data":"d0d924d27e1d42f3b666e76386101cf4afa1a8d207c244b9a4abc647bc099fff"} Nov 01 20:03:01.197314 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.197202 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hnrf" event={"ID":"4931755e-e9dc-4a2f-aab7-5d64dacb361e","Type":"ContainerStarted","Data":"0a4067d9ce3da41e2a1f026dad0705f5aaa2e71dff39186b7e891d5222ae612e"} Nov 01 20:03:01.197314 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.197226 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hnrf" event={"ID":"4931755e-e9dc-4a2f-aab7-5d64dacb361e","Type":"ContainerStarted","Data":"c67aaf4386b59b0c8251f4a342c210f399348d7264db59d377c2d88d4d07a756"} Nov 01 20:03:01.221165 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.221118 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:01.343910 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.343880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:01.383291 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.383183 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9hnrf" podStartSLOduration=2.578373428 podStartE2EDuration="5.383163094s" totalImagesPullingTime="2.721511927s" totalInitContainerRuntime="83.277739ms" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:01.381954629 +0000 UTC m=+61.992135580" watchObservedRunningTime="2026-11-01 20:03:01.383163094 +0000 UTC m=+61.993344107" Nov 01 20:03:01.499238 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.499196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:01.622025 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.621988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:01.746118 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.746086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:01.974317 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:01.974286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:02.052828 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.052796 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d569ccc7c-67pl4"] Nov 01 20:03:02.055550 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.055535 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.114012 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.113951 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 20:03:02.114012 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.113972 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 20:03:02.114012 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.113982 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 20:03:02.114545 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.114027 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6vk7hqa1puoju\"" Nov 01 20:03:02.114545 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.114061 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2r24h\"" Nov 01 20:03:02.114545 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.114225 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 20:03:02.115498 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.115478 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d569ccc7c-67pl4"] Nov 01 20:03:02.148132 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.148100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/853d6d5f-e795-45b9-b633-48f2f0051ffc-audit-log\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.148132 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.148143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-client-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.148369 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.148263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-client-certs\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.148369 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.148302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.148468 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.148420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-metrics-server-audit-profiles\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.148468 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.148460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpt9\" (UniqueName: \"kubernetes.io/projected/853d6d5f-e795-45b9-b633-48f2f0051ffc-kube-api-access-5tpt9\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.148540 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.148484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-tls\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.201674 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.201640 2573 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="712284bdce2a4f3561ca2a1b4288fc6a91da6ef981e35e8968dc028b7151ed58" exitCode=0 Nov 01 20:03:02.201835 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.201739 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"712284bdce2a4f3561ca2a1b4288fc6a91da6ef981e35e8968dc028b7151ed58"} Nov 01 20:03:02.249252 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.249149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-client-certs\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.249252 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.249235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.249521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.249294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-metrics-server-audit-profiles\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.249521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.249373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpt9\" (UniqueName: \"kubernetes.io/projected/853d6d5f-e795-45b9-b633-48f2f0051ffc-kube-api-access-5tpt9\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.249521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.249414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-tls\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.249521 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.249515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/853d6d5f-e795-45b9-b633-48f2f0051ffc-audit-log\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.249741 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.249559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-client-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.250263 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.250232 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.250494 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.250477 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/853d6d5f-e795-45b9-b633-48f2f0051ffc-audit-log\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.250648 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.250624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-metrics-server-audit-profiles\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.251904 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.251880 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-client-certs\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.252132 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.252117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-tls\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.252559 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.252540 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-client-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.305547 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.305514 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpt9\" (UniqueName: \"kubernetes.io/projected/853d6d5f-e795-45b9-b633-48f2f0051ffc-kube-api-access-5tpt9\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.366849 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.366807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:02.660761 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.660671 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9f57576c8-qm8gx"] Nov 01 20:03:02.664939 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.664913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.716970 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.716903 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 20:03:02.716970 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.716941 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 20:03:02.716970 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.716942 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 20:03:02.716970 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.716972 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 20:03:02.717322 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.717049 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 20:03:02.717322 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.717149 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sbflc\"" Nov 01 20:03:02.726990 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.726964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:02.753463 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/4d11ea05-bc33-4bbc-bcc3-942295c96058-kube-api-access-sdphh\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.753626 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753481 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.753626 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.753626 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.753760 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-federate-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.753760 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-metrics-client-ca\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.753826 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-serving-certs-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.753826 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.753809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.767202 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.767168 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 20:03:02.769157 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.769125 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9f57576c8-qm8gx"] Nov 01 20:03:02.783489 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.783448 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d569ccc7c-67pl4"] Nov 01 20:03:02.788898 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:03:02.788869 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853d6d5f_e795_45b9_b633_48f2f0051ffc.slice/crio-64bb6b442c763faca4d2f22c0e5cb351188f635cea7161488c120b63ac06bad2 WatchSource:0}: Error finding container 64bb6b442c763faca4d2f22c0e5cb351188f635cea7161488c120b63ac06bad2: Status 404 returned error can't find the container with id 64bb6b442c763faca4d2f22c0e5cb351188f635cea7161488c120b63ac06bad2 Nov 01 20:03:02.853780 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.853741 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:02.854819 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.854798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/4d11ea05-bc33-4bbc-bcc3-942295c96058-kube-api-access-sdphh\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.854869 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.854849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.854913 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.854867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.855017 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.854995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.855095 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.855074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-federate-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.855150 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.855132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-metrics-client-ca\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.855196 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.855160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-serving-certs-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.855252 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.855232 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.855983 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.855904 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.856098 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.856078 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-metrics-client-ca\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.856137 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.856099 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-serving-certs-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.857730 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.857697 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-federate-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.857828 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.857756 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.858213 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.858190 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.858295 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.858228 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.911257 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.911181 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/4d11ea05-bc33-4bbc-bcc3-942295c96058-kube-api-access-sdphh\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:02.980919 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:02.980881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:03.206147 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:03.206066 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" event={"ID":"853d6d5f-e795-45b9-b633-48f2f0051ffc","Type":"ContainerStarted","Data":"64bb6b442c763faca4d2f22c0e5cb351188f635cea7161488c120b63ac06bad2"} Nov 01 20:03:03.462709 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:03.462679 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9f57576c8-qm8gx"] Nov 01 20:03:03.464694 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:03:03.464666 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d11ea05_bc33_4bbc_bcc3_942295c96058.slice/crio-33094852f688fab1e75893cf2edfa817e212697837b2fe3d1511f8c2c42fea8e WatchSource:0}: Error finding container 33094852f688fab1e75893cf2edfa817e212697837b2fe3d1511f8c2c42fea8e: Status 404 returned error can't find the container with id 33094852f688fab1e75893cf2edfa817e212697837b2fe3d1511f8c2c42fea8e Nov 01 20:03:04.211190 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.211155 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" event={"ID":"4d11ea05-bc33-4bbc-bcc3-942295c96058","Type":"ContainerStarted","Data":"33094852f688fab1e75893cf2edfa817e212697837b2fe3d1511f8c2c42fea8e"} Nov 01 20:03:04.871544 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.871506 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:04.871783 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.871552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:03:04.922441 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.922405 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:03:04.922441 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.922432 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:03:04.935155 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.935120 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:04.937395 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.935133 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:03:04.973655 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:04.973610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:05.021862 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.021777 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" Nov 01 20:03:05.024160 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.024139 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:03:05.031261 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.031226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:05.084820 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.084501 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:03:05.091988 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.091959 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:05.217166 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.217130 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" event={"ID":"853d6d5f-e795-45b9-b633-48f2f0051ffc","Type":"ContainerStarted","Data":"3fe999076599c152d20f8121742cf6c1922ed942ca768b8b5399871bf2de5e7f"} Nov 01 20:03:05.220506 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.220481 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerStarted","Data":"e34ab782299ca56d7d3f3ec1bfb466251def8b2a08aa36e3763d86702b5cdde5"} Nov 01 20:03:05.220644 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.220536 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerStarted","Data":"8b47592d5a1847518a81aa57c20716ebceb0c55e44b57903cb6d3c84a76c82f4"} Nov 01 20:03:05.220644 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.220564 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerStarted","Data":"9cb53efa96168defab709ed4f51043ee051132eee735bc525927455717a3a938"} Nov 01 20:03:05.220644 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.220587 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerStarted","Data":"6f3a2e6476bc9704bcca2c44394a4a97e7e8cdea449f29bc995c7a9070b02e7f"} Nov 01 20:03:05.220644 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.220611 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerStarted","Data":"4a95e061e032a7c307d5a8be838537baacfeac76fd42a9c935c3a055c3a42723"} Nov 01 20:03:05.222273 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.222253 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" Nov 01 20:03:05.226655 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.226635 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" Nov 01 20:03:05.231688 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.231669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:05.236629 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.236497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:03:05.335276 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.335222 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" podStartSLOduration=1.407366045 podStartE2EDuration="3.335205337s" totalImagesPullingTime="1.927839292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:05.33442384 +0000 UTC m=+65.944604806" watchObservedRunningTime="2026-11-01 20:03:05.335205337 +0000 UTC m=+65.945386293" Nov 01 20:03:05.446638 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.444384 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7lwmf"] Nov 01 20:03:05.448105 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:03:05.447880 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6b3e1d_ddca_43d2_b5c4_300dcacd99ee.slice/crio-1a144e458a6f1fdf6107ebae05d781f99e384e1f9282c814aed8bcf4c359c046 WatchSource:0}: Error finding container 1a144e458a6f1fdf6107ebae05d781f99e384e1f9282c814aed8bcf4c359c046: Status 404 returned error can't find the container with id 1a144e458a6f1fdf6107ebae05d781f99e384e1f9282c814aed8bcf4c359c046 Nov 01 20:03:05.608147 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.608064 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ks9th"] Nov 01 20:03:05.612011 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:03:05.611976 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7737ff_1496_4eed_9bfb_b3e6ea260132.slice/crio-2d5db8c010ce1bc69b6ac71fdf7325bc1072b77aba4a1865db05280e12a3717a WatchSource:0}: Error finding container 2d5db8c010ce1bc69b6ac71fdf7325bc1072b77aba4a1865db05280e12a3717a: Status 404 returned error can't find the container with id 2d5db8c010ce1bc69b6ac71fdf7325bc1072b77aba4a1865db05280e12a3717a Nov 01 20:03:05.616563 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:05.616482 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qm6s6"] Nov 01 20:03:05.619938 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:03:05.619913 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4561dd90_478e_46b0_805e_b2ead5252a83.slice/crio-bc4fc9851fd62b181e86b1bef2ec07dea07783ffe2e17ab12fcc0169beb8cc6e WatchSource:0}: Error finding container bc4fc9851fd62b181e86b1bef2ec07dea07783ffe2e17ab12fcc0169beb8cc6e: Status 404 returned error can't find the container with id bc4fc9851fd62b181e86b1bef2ec07dea07783ffe2e17ab12fcc0169beb8cc6e Nov 01 20:03:06.011232 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.011204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:06.052087 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.052049 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:06.056246 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.056220 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082255 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082483 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjxds\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082483 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082483 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082646 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082646 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082646 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082795 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082795 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082795 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082795 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.082795 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083031 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083031 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083031 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083031 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.082908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083031 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.083025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083237 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.083070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083237 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.083124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.083237 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.083155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.109161 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.108820 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 20:03:06.109161 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.108882 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 20:03:06.109161 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.109062 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 20:03:06.109161 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.109109 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 20:03:06.109161 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.109148 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 20:03:06.109161 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.109167 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 20:03:06.109627 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.109181 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ft87h\"" Nov 01 20:03:06.109627 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.109187 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 20:03:06.109627 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.109222 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 20:03:06.157862 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.157706 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 20:03:06.157862 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.157749 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 20:03:06.157862 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.157749 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 20:03:06.157862 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.157797 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 20:03:06.158128 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.157869 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c9o03dg9n84k5\"" Nov 01 20:03:06.184377 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184559 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184559 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184559 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184559 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184559 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184559 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjxds\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.184854 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.185289 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.185289 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.184912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.185289 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.185080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.186058 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.185749 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.187092 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.186431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.187092 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.186735 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.187092 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.186901 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.188068 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.188039 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.188697 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.188226 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.188697 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.188307 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.189274 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.189164 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.190121 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.189860 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.190121 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.189982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.190608 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.190528 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.190928 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.190908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.191139 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.191102 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.191872 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.191830 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.192383 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.192310 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.192474 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.192437 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.192777 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.192757 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.208870 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.208837 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 20:03:06.209195 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.209173 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 20:03:06.210597 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.210514 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:06.217476 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.217449 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.218128 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.218097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.224981 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.224955 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qm6s6" event={"ID":"4561dd90-478e-46b0-805e-b2ead5252a83","Type":"ContainerStarted","Data":"bc4fc9851fd62b181e86b1bef2ec07dea07783ffe2e17ab12fcc0169beb8cc6e"} Nov 01 20:03:06.228252 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.228229 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerStarted","Data":"960b1bfed035003faab0ff12c7192c3b87fa0fa54c42cb78f38144a522dd5488"} Nov 01 20:03:06.229461 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.229437 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7lwmf" event={"ID":"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee","Type":"ContainerStarted","Data":"1a144e458a6f1fdf6107ebae05d781f99e384e1f9282c814aed8bcf4c359c046"} Nov 01 20:03:06.230538 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.230515 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ks9th" event={"ID":"5f7737ff-1496-4eed-9bfb-b3e6ea260132","Type":"ContainerStarted","Data":"2d5db8c010ce1bc69b6ac71fdf7325bc1072b77aba4a1865db05280e12a3717a"} Nov 01 20:03:06.259531 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.259503 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjxds\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.358500 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.358447 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.541320276 podStartE2EDuration="7.358432859s" totalImagesPullingTime="4.802975271s" totalInitContainerRuntime="14.137312ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:06.357417228 +0000 UTC m=+66.967598218" watchObservedRunningTime="2026-11-01 20:03:06.358432859 +0000 UTC m=+66.968613805" Nov 01 20:03:06.387211 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.387159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:06.826041 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:06.826009 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:06.831243 ip-10-0-128-78 kubenswrapper[2573]: W1101 20:03:06.831204 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fab5652_b1da_42d8_9c42_1e32ad4fc054.slice/crio-7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add WatchSource:0}: Error finding container 7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add: Status 404 returned error can't find the container with id 7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add Nov 01 20:03:07.239731 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:07.239692 2573 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" exitCode=0 Nov 01 20:03:07.240818 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:07.240791 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9"} Nov 01 20:03:07.240947 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:07.240852 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerStarted","Data":"7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add"} Nov 01 20:03:07.251020 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:07.250916 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:07.374314 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:07.374275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:07.518356 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:07.518288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:07.646536 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:07.646505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:08.115984 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:08.115917 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:08.236577 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:08.236538 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:08.247621 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:08.247583 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qm6s6" event={"ID":"4561dd90-478e-46b0-805e-b2ead5252a83","Type":"ContainerStarted","Data":"e31c00f4e76fb14299c90b214d4e5fe0c7ce71703950edf17c43567e6bfc4769"} Nov 01 20:03:08.248090 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:08.247651 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qm6s6" event={"ID":"4561dd90-478e-46b0-805e-b2ead5252a83","Type":"ContainerStarted","Data":"79b3c17f6d648ef5d09cbb899e01ddf45e2e14936ebd4ada31212094ba71665b"} Nov 01 20:03:08.360312 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:08.360250 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qm6s6" podStartSLOduration=66.551495352 podStartE2EDuration="1m8.360233324s" totalImagesPullingTime="1.808737972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:08.359697392 +0000 UTC m=+68.969878341" watchObservedRunningTime="2026-11-01 20:03:08.360233324 +0000 UTC m=+68.970414269" Nov 01 20:03:09.480557 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:09.480489 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:09.601774 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:09.601739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:10.167257 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:10.167221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:10.261398 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:10.261356 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ks9th" event={"ID":"5f7737ff-1496-4eed-9bfb-b3e6ea260132","Type":"ContainerStarted","Data":"cc03eec3a60e61bedd53ec7dde35fab994c958183386cc96911dec2f50cfe2c3"} Nov 01 20:03:10.262157 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:10.262134 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:10.368415 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:10.368372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:10.461232 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:10.461178 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ks9th" podStartSLOduration=65.950164321 podStartE2EDuration="1m10.461163714s" totalImagesPullingTime="4.510999393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:10.459888699 +0000 UTC m=+71.070069670" watchObservedRunningTime="2026-11-01 20:03:10.461163714 +0000 UTC m=+71.071344660" Nov 01 20:03:11.268651 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:11.268610 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" event={"ID":"4d11ea05-bc33-4bbc-bcc3-942295c96058","Type":"ContainerStarted","Data":"ee6ac6b6d8af93f1fb69eb825df6b62c7dd6c9d394d2d9e2e6f44196715131ee"} Nov 01 20:03:11.269117 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:11.268685 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" event={"ID":"4d11ea05-bc33-4bbc-bcc3-942295c96058","Type":"ContainerStarted","Data":"9968e2bcbde7315cdc34e4a3b929666fced84d6e58221aca06693e2c0d5d0c5e"} Nov 01 20:03:11.269117 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:11.268718 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" event={"ID":"4d11ea05-bc33-4bbc-bcc3-942295c96058","Type":"ContainerStarted","Data":"28953b812e943ebad5fedba0c5d22269a7f8edc7a0f95bcf7d2a5a5aa280adfa"} Nov 01 20:03:11.391934 ip-10-0-128-78 kubenswrapper[2573]: I1101 20:03:11.391490 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" podStartSLOduration=2.379142884 podStartE2EDuration="9.391470088s" totalImagesPullingTime="7.012327204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:11.390539825 +0000 UTC m=+72.000720774" watchObservedRunningTime="2026-11-01 20:03:11.391470088 +0000 UTC m=+72.001651050" Nov 01 20:03:12.264436 ip-10-0-128-78 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 20:03:12.282789 ip-10-0-128-78 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 20:03:12.282973 ip-10-0-128-78 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 20:03:12.283026 ip-10-0-128-78 systemd[1]: kubelet.service: Consumed 4.545s CPU time. Nov 01 20:03:12.292045 ip-10-0-128-78 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:03:12.361035 ip-10-0-128-78 kubenswrapper[5633]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:12.361035 ip-10-0-128-78 kubenswrapper[5633]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:03:12.361035 ip-10-0-128-78 kubenswrapper[5633]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:12.361035 ip-10-0-128-78 kubenswrapper[5633]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:12.363233 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363141 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.363233 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363232 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.363233 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363237 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363240 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363243 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363247 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363249 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363259 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363262 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363265 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363267 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363270 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363273 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363276 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363279 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363281 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363284 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363287 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363289 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363292 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363294 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363297 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.363353 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363299 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363302 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363304 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363307 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363310 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363314 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363319 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363322 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363339 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363343 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363346 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363349 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363352 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363354 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363357 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363360 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363362 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363379 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363382 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.363977 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363385 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363387 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363390 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363392 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363395 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363398 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363400 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363403 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363406 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363409 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363411 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363414 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363417 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363420 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363422 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363425 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363430 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363434 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363437 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.364525 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363439 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363442 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363444 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363447 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363450 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363453 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363456 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363459 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363461 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363464 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363466 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363469 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363472 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363475 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363477 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363480 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363482 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363485 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363487 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363490 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.364989 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363492 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363495 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363497 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363500 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363503 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363505 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363508 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363511 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363513 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363516 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363519 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363521 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363524 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363526 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363529 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363532 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363537 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363539 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363542 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363545 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.365486 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363548 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363550 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363553 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363555 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363558 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.363562 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364572 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364582 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364585 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364588 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364591 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364593 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364596 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364599 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364601 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364604 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364606 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364609 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364617 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.365964 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364620 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364622 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364625 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364628 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364631 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364634 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364636 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364639 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364642 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364645 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364648 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364651 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364653 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364656 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364658 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364660 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364663 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364666 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364668 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364672 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.366443 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364674 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364677 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364679 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364681 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364684 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364686 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364689 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364691 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364694 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364697 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364702 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364706 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364709 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364713 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364715 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364718 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364720 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364723 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364726 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.366921 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364728 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364730 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364733 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364737 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364740 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364742 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364745 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364748 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364750 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364753 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364755 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364758 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364760 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364763 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364766 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364769 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364771 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364773 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364776 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364778 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.367390 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364782 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364784 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364787 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364790 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364793 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364796 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364798 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364801 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364803 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364806 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364808 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364811 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364813 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364816 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364818 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364821 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364824 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364826 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364828 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.367874 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364832 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364836 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364839 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364841 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364844 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364846 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364849 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364854 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364857 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364860 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364862 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364865 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364867 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364870 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.364873 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365404 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365410 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365413 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365416 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365418 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.368352 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365421 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365423 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365426 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365430 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365433 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365436 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365438 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365441 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365444 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365448 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365451 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365453 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365456 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365458 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365461 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365463 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365466 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365468 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365471 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.368847 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365474 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365477 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365479 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365482 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365484 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365487 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365490 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365492 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365495 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365499 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365502 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365504 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365507 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365509 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365512 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365514 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365517 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365520 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365523 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365525 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.369397 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365528 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365531 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365534 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365537 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365540 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365542 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365545 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365547 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365550 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365552 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365555 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365557 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365560 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365563 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365565 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365568 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365570 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365574 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365576 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365579 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.369974 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365581 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365584 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365588 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365592 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365595 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365597 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365600 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365603 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365605 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365607 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365610 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365612 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365615 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365617 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365621 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365624 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365626 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365629 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365632 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.370755 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365634 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365637 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365639 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365642 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365644 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365647 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365649 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365652 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365655 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365658 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365660 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365663 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365665 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365668 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365672 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365674 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365677 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365679 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365682 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365684 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.371264 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365687 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365689 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.365693 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365799 5633 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365807 5633 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365813 5633 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365819 5633 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365825 5633 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365829 5633 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365835 5633 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365840 5633 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365843 5633 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365846 5633 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365850 5633 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365853 5633 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365856 5633 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365859 5633 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365862 5633 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365865 5633 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365868 5633 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365872 5633 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365877 5633 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365880 5633 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365883 5633 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:03:12.371766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365886 5633 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365889 5633 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365894 5633 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365897 5633 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365901 5633 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365904 5633 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365907 5633 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365910 5633 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365913 5633 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365916 5633 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365919 5633 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365923 5633 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365926 5633 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365929 5633 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365932 5633 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365937 5633 flags.go:64] FLAG: --enable-server="true" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365940 5633 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365944 5633 flags.go:64] FLAG: --event-burst="100" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365947 5633 flags.go:64] FLAG: --event-qps="50" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365950 5633 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365953 5633 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365956 5633 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365960 5633 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365963 5633 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365967 5633 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:03:12.372356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365970 5633 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365973 5633 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365976 5633 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365979 5633 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365983 5633 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365986 5633 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365988 5633 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365991 5633 flags.go:64] FLAG: --feature-gates="" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.365995 5633 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366000 5633 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366003 5633 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366006 5633 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366009 5633 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366013 5633 flags.go:64] FLAG: --help="false" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366016 5633 flags.go:64] FLAG: --hostname-override="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366019 5633 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366022 5633 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366025 5633 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366028 5633 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366032 5633 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366035 5633 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366038 5633 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366041 5633 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:03:12.372979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366045 5633 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366047 5633 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366051 5633 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366054 5633 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366057 5633 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366059 5633 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366063 5633 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366065 5633 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366068 5633 flags.go:64] FLAG: --lock-file="" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366071 5633 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366074 5633 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366077 5633 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366082 5633 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366085 5633 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366088 5633 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366091 5633 flags.go:64] FLAG: --logging-format="text" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366094 5633 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366098 5633 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366102 5633 flags.go:64] FLAG: --manifest-url="" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366105 5633 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366109 5633 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366113 5633 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366117 5633 flags.go:64] FLAG: --max-pods="110" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366120 5633 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366123 5633 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:03:12.373555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366126 5633 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366128 5633 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366131 5633 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366134 5633 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366137 5633 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366144 5633 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366149 5633 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366152 5633 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366156 5633 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366159 5633 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366161 5633 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366164 5633 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366168 5633 flags.go:64] FLAG: --port="10250" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366171 5633 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366173 5633 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-005c246d98ad74763" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366176 5633 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366179 5633 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366182 5633 flags.go:64] FLAG: --register-node="true" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366185 5633 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366189 5633 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366192 5633 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366195 5633 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366198 5633 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366202 5633 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366205 5633 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366211 5633 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:03:12.374149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366213 5633 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366216 5633 flags.go:64] FLAG: --runonce="false" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366219 5633 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366222 5633 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366225 5633 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366228 5633 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366231 5633 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366234 5633 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366237 5633 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366241 5633 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366243 5633 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366246 5633 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366250 5633 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366253 5633 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366256 5633 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366259 5633 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366262 5633 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366267 5633 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366270 5633 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366273 5633 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366277 5633 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366280 5633 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366283 5633 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366286 5633 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366289 5633 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:03:12.374830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366292 5633 flags.go:64] FLAG: --v="2" Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366296 5633 flags.go:64] FLAG: --version="false" Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366300 5633 flags.go:64] FLAG: --vmodule="" Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366304 5633 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366308 5633 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366483 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366488 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366491 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366494 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366497 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366499 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366501 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366504 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366507 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366509 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366512 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366514 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366517 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366519 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366523 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.375468 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366526 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366528 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366531 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366533 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366536 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366538 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366541 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366543 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366546 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366548 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366551 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366553 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366556 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366558 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366561 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366563 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366566 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366569 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366571 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366574 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.375942 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366576 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366581 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366584 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366587 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366590 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366593 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366596 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366598 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366601 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366604 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366606 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366611 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366613 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366615 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366618 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366621 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366623 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366626 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366628 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.376535 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366631 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366633 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366635 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366638 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366641 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366643 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366646 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366648 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366651 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366653 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366656 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366659 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366662 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366665 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366667 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366670 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366672 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366675 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366677 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366679 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.377160 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366682 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366684 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366687 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366690 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366693 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366696 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366698 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366700 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366703 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366706 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366708 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366711 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366714 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366718 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366720 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366723 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366725 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366728 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366731 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.377854 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366734 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366736 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366739 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366741 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366744 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366747 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366749 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366752 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366754 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366757 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366759 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366762 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366764 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.366790 5633 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366828 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366832 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.378477 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366835 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366838 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366842 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366844 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366847 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366849 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366852 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366855 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366857 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366860 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366862 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366865 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366867 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366870 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366872 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366874 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366878 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366881 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366883 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366886 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.378872 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366888 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366891 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366894 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366897 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366901 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366905 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366908 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366910 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366913 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366915 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366918 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366920 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366922 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366925 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366929 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366931 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366935 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366937 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366940 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.379367 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366942 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366945 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366947 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366950 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366952 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366955 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366957 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366960 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366962 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366966 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366968 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366971 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366974 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366976 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366979 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366981 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366985 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366987 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366991 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366993 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.379963 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366995 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.366998 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367000 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367003 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367005 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367008 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367010 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367013 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367016 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367018 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367021 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367023 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367026 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367028 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367032 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367035 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367038 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367041 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367043 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.380695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367045 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367048 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367052 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367055 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367057 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367059 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367062 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367064 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367067 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367070 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367072 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367075 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367077 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367080 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367082 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367085 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367087 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367089 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367092 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.381258 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367094 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367097 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367099 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367101 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367105 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367107 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.367110 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.372214 5633 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.372234 5633 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.372254 5633 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.372260 5633 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372322 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372346 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372352 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372357 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372361 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372364 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372367 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372372 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.381826 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372375 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372379 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372381 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372393 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372397 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372400 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372404 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372407 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372410 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372413 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372416 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372419 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372421 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372424 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372427 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372430 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372433 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372436 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372438 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.382284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372441 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372443 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372447 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372450 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372453 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372455 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372458 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372460 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372463 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372465 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372468 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372471 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372473 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372476 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372478 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372480 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372483 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372486 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372488 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372491 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.382782 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372494 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372497 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372500 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372503 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372505 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372508 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372511 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372513 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372516 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372518 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372521 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372523 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372526 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372528 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372531 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372533 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372537 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372541 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372545 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.383277 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372547 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372550 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372553 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372555 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372557 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372560 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372563 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372565 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372567 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372570 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372572 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372575 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372577 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372581 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372584 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372586 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372589 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372591 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372594 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372596 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.383813 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372600 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372602 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372605 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372607 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372610 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372612 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372615 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372617 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372619 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372622 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372625 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372627 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372629 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372632 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372634 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372637 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372639 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372642 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372644 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.384304 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372646 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.372678 5633 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372834 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372839 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372842 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372844 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372847 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372850 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372853 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372856 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372859 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372861 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372864 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372867 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372869 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372872 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.384923 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372874 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372877 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372880 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372882 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372885 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372887 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372890 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372892 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372895 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372897 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372899 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372902 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372905 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372907 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372910 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372912 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372915 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372917 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372921 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.385761 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372924 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372927 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372930 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372932 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372935 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372938 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372941 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372944 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372946 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372949 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372952 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372954 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372957 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372959 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372962 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372964 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372967 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372969 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372972 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372974 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.386247 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372977 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372979 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372982 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372985 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372988 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372990 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372993 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372995 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.372998 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373000 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373003 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373005 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373008 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373010 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373013 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373015 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373018 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373020 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373023 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.386844 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373025 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373028 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373030 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373032 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373035 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373038 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373040 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373043 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373045 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373048 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373050 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373053 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373055 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373058 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373060 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373063 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373065 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373068 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373070 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373072 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.387396 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373075 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373078 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373080 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373083 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373086 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373089 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373093 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373096 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373099 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373102 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373104 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373107 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373109 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373112 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.373136 5633 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:12.387978 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373166 5633 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373171 5633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373173 5633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373176 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373179 5633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373182 5633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373184 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373187 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373189 5633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373192 5633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373194 5633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373197 5633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373199 5633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373202 5633 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373204 5633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373207 5633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373210 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373212 5633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373215 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:12.388415 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373217 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373221 5633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373225 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373227 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373230 5633 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373232 5633 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373235 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373237 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373240 5633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373242 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373245 5633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373248 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373250 5633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373253 5633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373255 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373257 5633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373260 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373262 5633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373265 5633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:12.389013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373267 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373269 5633 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373272 5633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373274 5633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373277 5633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373279 5633 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373282 5633 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373284 5633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373287 5633 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373289 5633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373292 5633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373294 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373297 5633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373299 5633 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373302 5633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373304 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373307 5633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373309 5633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373312 5633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373314 5633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:12.389651 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373317 5633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373319 5633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373321 5633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373342 5633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373347 5633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373351 5633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373354 5633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373356 5633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373359 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373361 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373363 5633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373366 5633 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373369 5633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373372 5633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373374 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373377 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373379 5633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373381 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373384 5633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373386 5633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:12.390284 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373389 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373391 5633 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373394 5633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373396 5633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373399 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373401 5633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373404 5633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373406 5633 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373409 5633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373411 5633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373414 5633 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373416 5633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373419 5633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373421 5633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373423 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373426 5633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373428 5633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373431 5633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373435 5633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373437 5633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:12.390952 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373440 5633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373442 5633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373444 5633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373447 5633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373449 5633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373451 5633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373454 5633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:12.373461 5633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.373523 5633 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.374534 5633 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.374597 5633 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.374776 5633 server.go:1014] "Starting client certificate rotation" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.374916 5633 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.374960 5633 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:57:00 +0000 UTC" deadline="2028-06-11 00:39:31.829218444 +0000 UTC" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.374988 5633 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14092h36m19.454232784s" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.375233 5633 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:03:12.391514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.376321 5633 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:03:12.392069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.377652 5633 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:03:12.392069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.379602 5633 log.go:25] "Validated CRI v1 image API" Nov 01 20:03:12.392069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.380356 5633 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:03:12.392069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.382678 5633 fs.go:135] Filesystem UUIDs: map[52e88fd5-fee8-4410-b6e7-195d5680e98a:/dev/nvme0n1p4 678924c3-63db-47a3-a938-c326d17f0743:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 01 20:03:12.392311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.382698 5633 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a144e458a6f1fdf6107ebae05d781f99e384e1f9282c814aed8bcf4c359c046/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a144e458a6f1fdf6107ebae05d781f99e384e1f9282c814aed8bcf4c359c046/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d5db8c010ce1bc69b6ac71fdf7325bc1072b77aba4a1865db05280e12a3717a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d5db8c010ce1bc69b6ac71fdf7325bc1072b77aba4a1865db05280e12a3717a/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33094852f688fab1e75893cf2edfa817e212697837b2fe3d1511f8c2c42fea8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33094852f688fab1e75893cf2edfa817e212697837b2fe3d1511f8c2c42fea8e/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ac77269f540a824c3bb9ac53fe976f5a6ce8f4178f75492ff12ae62b9ba3d35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ac77269f540a824c3bb9ac53fe976f5a6ce8f4178f75492ff12ae62b9ba3d35/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5464920b79c38dbf826a9b3bc04dc33a62856daa653a58683209e26c50864fa5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5464920b79c38dbf826a9b3bc04dc33a62856daa653a58683209e26c50864fa5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/615cde627b6896c432dd15a2527790a8550808513d6fb140f101c7cae202ce09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/615cde627b6896c432dd15a2527790a8550808513d6fb140f101c7cae202ce09/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64bb6b442c763faca4d2f22c0e5cb351188f635cea7161488c120b63ac06bad2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64bb6b442c763faca4d2f22c0e5cb351188f635cea7161488c120b63ac06bad2/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67304fd16b169c449dc4c31ef68631157af0ed6d0956b91f961f0ca09993cc26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67304fd16b169c449dc4c31ef68631157af0ed6d0956b91f961f0ca09993cc26/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8658d7e408b8697cf601f810e2d9ca3c27e1f0eb5e2d88020910a2d9b6f8fe69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8658d7e408b8697cf601f810e2d9ca3c27e1f0eb5e2d88020910a2d9b6f8fe69/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c049b5fae3b06e7b163dcb50beff63b937223e0c301b4b9421c7b8664d65d7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c049b5fae3b06e7b163dcb50beff63b937223e0c301b4b9421c7b8664d65d7b/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd/userdata/shm major:0 minor:65 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fedaa5297a99a3c3a39916c21582943e803408a4355d5943634acee92b277de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fedaa5297a99a3c3a39916c21582943e803408a4355d5943634acee92b277de/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4c93e2c40fb25c207514420be7c29cb149d1b07952fdd6d50bbb2478a9b0831/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4c93e2c40fb25c207514420be7c29cb149d1b07952fdd6d50bbb2478a9b0831/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2a333f0c31ea725139b310d3b8ca0a0833f522d72ee683aa320c99d9b213699/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2a333f0c31ea725139b310d3b8ca0a0833f522d72ee683aa320c99d9b213699/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc4fc9851fd62b181e86b1bef2ec07dea07783ffe2e17ab12fcc0169beb8cc6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc4fc9851fd62b181e86b1bef2ec07dea07783ffe2e17ab12fcc0169beb8cc6e/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7725c3b6cf4533801c588c89317e48d70b3a0850879a957c0794e5146418e84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7725c3b6cf4533801c588c89317e48d70b3a0850879a957c0794e5146418e84/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d40959f94916497c751f22e6b9995f21018fe0fda22912ebf7952e5c43ae4433/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d40959f94916497c751f22e6b9995f21018fe0fda22912ebf7952e5c43ae4433/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db9c2d78f93b4e895f0749ac63cfecd5081b4a2d2387199c42117ca83b46853d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db9c2d78f93b4e895f0749ac63cfecd5081b4a2d2387199c42117ca83b46853d/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc2fb59201257eefec5a5fbbe040c89fc1e0cb7eaad7ad6480f8b3b1395aacb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc2fb59201257eefec5a5fbbe040c89fc1e0cb7eaad7ad6480f8b3b1395aacb7/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0d3a8b55cb40dfd470791821223c8cbbd9f5f27ffc5ab3e2feb8d5aa80e8396/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0d3a8b55cb40dfd470791821223c8cbbd9f5f27ffc5ab3e2feb8d5aa80e8396/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0db527d3-b994-4c15-9a6e-3c01ea4b092e/volumes/kubernetes.io~projected/kube-api-access-s5t6p:{mountpoint:/var/lib/kubelet/pods/0db527d3-b994-4c15-9a6e-3c01ea4b092e/volumes/kubernetes.io~projected/kube-api-access-s5t6p major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/config-reloader/2 major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/prometheus/15 major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~empty-dir/config-out major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~projected/kube-api-access-gjxds:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~projected/kube-api-access-gjxds major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~projected/tls-assets major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/config major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/web-config major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a5a4a2b-455d-42ac-8064-cb987052b764/volumes/kubernetes.io~projected/kube-api-access-frr7p:{mountpoint:/var/lib/kubelet/pods/3a5a4a2b-455d-42ac-8064-cb987052b764/volumes/kubernetes.io~projected/kube-api-access-frr7p major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef/volumes/kubernetes.io~projected/kube-api-access-ljcc7:{mountpoint:/var/lib/kubelet/pods/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef/volumes/kubernetes.io~projected/kube-api-access-ljcc7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4561dd90-478e-46b0-805e-b2ead5252a83/volumes/kubernetes.io~projected/kube-api-access-hgm5b:{mountpoint:/var/lib/kubelet/pods/4561dd90-478e-46b0-805e-b2ead5252a83/volumes/kubernetes.io~projected/kube-api-access-hgm5b major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4561dd90-478e-46b0-805e-b2ead5252a83/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4561dd90-478e-46b0-805e-b2ead5252a83/volumes/kubernetes.io~secret/metrics-certs major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/460d8bce-a4b3-4a43-b5cb-f7ff32651690/volumes/kubernetes.io~projected/kube-api-access-647zs:{mountpoint:/var/lib/kubelet/pods/460d8bce-a4b3-4a43-b5cb-f7ff32651690/volumes/kubernetes.io~projected/kube-api-access-647zs major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~projected/kube-api-access-np598:{mountpoint:/var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~projected/kube-api-access-np598 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~projected/kube-api-access-sdphh:{mountpoint:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~projected/kube-api-access-sdphh major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5931bef0-3ad6-4758-82c5-352add308c77/volumes/kubernetes.io~projected/kube-api-access-tg7pl:{mountpoint:/var/lib/kubelet/pods/5931bef0-3ad6-4758-82c5-352add308c77/volumes/kubernetes.io~projected/kube-api-access-tg7pl major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f7737ff-1496-4eed-9bfb-b3e6ea260132/volumes/kubernetes.io~projected/kube-api-access-2rkgr:{mountpoint:/var/lib/kubelet/pods/5f7737ff-1496-4eed-9bfb-b3e6ea260132/volumes/kubernetes.io~projected/kube-api-access-2rkgr major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~projected/kube-api-access-5tpt9:{mountpoint:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~projected/kube-api-access-5tpt9 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~projected/kube-api-access-rv69v:{mountpoint:/var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~projected/kube-api-access-rv69v major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/alertmanager/9 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/config-reloader/6 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~projected/kube-api-access-qfggb:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~projected/kube-api-access-qfggb major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~projected/tls-assets major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/config-volume major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/web-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~projected/kube-api-access-cjlgj:{mountpoint:/var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~projected/kube-api-access-cjlgj major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93e86161-3b78-4cd6-960d-70755fcc7b76/volumes/kubernetes.io~projected/kube-api-access-jm4zh:{mountpoint:/var/lib/kubelet/pods/93e86161-3b78-4cd6-960d-70755fcc7b76/volumes/kubernetes.io~projected/kube-api-access-jm4zh major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93e86161-3b78-4cd6-960d-70755fcc7b76/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/93e86161-3b78-4cd6-960d-70755fcc7b76/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volumes/kubernetes.io~projected/kube-api-access-rlhh8:{mountpoint:/var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volumes/kubernetes.io~projected/kube-api-access-rlhh8 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5/volumes/kubernetes.io~projected/kube-api-access-mlt72:{mountpoint:/var/lib/kubelet/pods/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5/volumes/kubernetes.io~projected/kube-api-access-mlt72 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d979ea87-f591-49d1-a326-bb58c0af72aa/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d979ea87-f591-49d1-a326-bb58c0af72aa/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e86d2ca4-1f79-454b-bd82-e86db21a1abc/volumes/kubernetes.io~projected/kube-api-access-4wmnv:{mountpoint:/var/lib/kubelet/pods/e86d2ca4-1f79-454b-bd82-e86db21a1abc/volumes/kubernetes.io~projected/kube-api-access-4wmnv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e86d2ca4-1f79-454b-bd82-e86db21a1abc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e86d2ca4-1f79-454b-bd82-e86db21a1abc/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0851759-372d-4706-8bf7-ee6a493872f4/volumes/kubernetes.io~projected/kube-api-access-rqtwv:{mountpoint:/var/lib/kubelet/pods/f0851759-372d-4706-8bf7-ee6a493872f4/volumes/kubernetes.io~projected/kube-api-access-rqtwv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0851759-372d-4706-8bf7-ee6a493872f4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f0851759-372d-4706-8bf7-ee6a493872f4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~projected/kube-api-access-7shpz:{mountpoint:/var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~projected/kube-api-access-7shpz major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:275 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/270bc4488cc6dc98c547a3f25e86d97c104d3781b6ef86769ae0ecef99339fcb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f4fcbe28eb9ed02695e241c9082be9a147844dba3949c95f56aac7c45985c27c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/aa3c3fdf087a0e1885593b40a33e3a57e2ca618ce46b5a86fabd8b9d29ae72fe/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1b8ad5fa4d781f8642d37131e7eb1d845933d4aa7003d3a1e85a274a970ebc18/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a5ee554c44b416d81c17d65a9dc59b325de9c25c1737002537ec201089214897/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/26fa49419e82354a9253e4f295b5d29fbee057833952d6f1d0a62808d3f8bc9e/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0f5af906b00d773fcef1a5c8213a8980d7b082997d3b7f115b52b195521d0d4e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f65cb5c75baadcd60d471f939c4f6a29320470e04ee9fca1bd352cc936bc486a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5fbb8aee624193e03368820c4ee74fd822c5eadd47710ecd7c75788ed5872d4f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ddb7fb7b553ca352c1810a1dbd8f2395eb34921b5888c9b96d49a3221acf107f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5aa3c5f28c65244200cb17001d7a42b277379a4aa5969ba269c04efde04c572f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/c9dae65228bd8b4cd2223b282b5808d597a2c4489e5dccdf908b734104d273ca/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f7f5ac78bb304dd266496a6f3f2e1a834347c205291d76c42c639ab6695657fe/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5683e58b15f66e2c7594792831a87ac47f827dd3a6accb1905ed2cb983d64656/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9958350187b071286adf5569d541ab1b7fc512bbe3f088c1ee62d7235a40267b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/44e200e764e1db3e284f45a6803ec502d915bb1994ee6581a8a18ea0a0542a07/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a56d6a503b0372c60c39a1df3d001762c2c41ba3f3232a1791b25f5c41e34bfa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/db98ad81bd216ff7242f1ebeefaf3cb77a6fbfb6c327d0f272b012f8dd7d0aa8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/15e901aa51816e83f363d223792df1550267bed93d577c01579104da61656564/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/806921b1e4168ca47e874154809f9eb7323d50c22db9b5e7a9f4ec7e2aafefda/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/587d43d0281a1ddb3ff831febc735b4d618b4a7007a564dfe35316e5ce5c9b4a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/08d077b19fbd5e96432031915a564f227e74e92545e0308903bb704f6a08b701/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/874fc80bed4cbf0aed51f679af0b133bbb17bedff6d3c1d3a1c23f70f2dc18f9/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/ba551585f35c543c502eca2772cf22f1058007c5c4122e2cd648110fe520a402/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d23292297c604d66964c44cb90b615613ac6dcc3e81cea7ebaa0bff75b8d242e/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b88fc2704e7f8f4f607b695e38da4364f1ac00389cd584997cb3675027b1833e/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/a91d47a0de2910b68b5aabc366197c9a1baf3639e513701dc10c30e468c37b58/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/2b351b1f146474ea672ded48c8f954fd37050a82228abafc34775f3ed2757102/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/9c3b4c34ed8251acbacab7b71d3728cc608a26fbd9211898ea88bd8097e1ae41/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/0a24fed13a908f326cec6152339c00feefbba65c9ef3ff9baf9ea1e1ea71cfad/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/0ca46893068acb1abc2c050329e8152b907e8e231bca3421894893874e217a1e/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/d57585e5590e339e72d01541b02a2d5f2c705e08558daa5a4a527885f3aaec9e/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/6327f59590598407e45ba361ff152dbda18b20da2c7cb2b56682fd79f366730a/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/eee0447b80c74fecfe6de5ab36a74c39ec321f53b2867de2ae5546c984e5e6ef/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/9fe62df2055e27fa3a2669c983ad23794f6ef4f3e84bed01ea3be65c81527e02/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/6f0367dc240a7fc51096bd919b0828f949e8a64d0c0c8f226b4c6cfe4d7a2a76/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/3136fa5aac9d733d3ee1ffa6ffc116b130993ec8b2b15be8a1a26bb3f110562c/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/1a01bf9a9fe5d1f51531466ff0e1cfa1c460d6d1a7a010c0a3b14ebef905c35d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e92b5b503ed5d7d6a2e5a4802a5c2827f2cc2fd0e2edbe63928ecdeb51a66f1d/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/8a3a8e87e15174f6eab7c90b25ce3c6af4a1f5ce323d11b49fd891cf6f58b5bf/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/90229dc70e75021f7508432124aa294fe06c83f84adb3e6b43bbbc08b98044d7/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/5fd7b4f08885d49a0f734edab613439fe27b9b0235ced757fb137c91aa8fa40a/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/34dad9928abb0c63c02b3736945cf59c203305e053086d4994ee3fadff554ce1/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/3e6758405c1ff496dd5fa57c9a8bc1c08325654dc541b77be8d836974a2dd75c/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/3fdf6eb212b9f1400c284b14af56585dd106f1070c489bdc60600dcf42e443a3/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/58354cc429ca6db63e4bd922bc1ee27f6219ee1da4e76eee37abd4a0a5168371/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/733abc06cec3f4c3a733b5d3a075528fe4f2b6c92cdbaa31ce508d1f1bedc010/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4819cd2c675d182a6e1add071b8c62ad217cea19260d89c3113559016b4d068e/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/166981acef1827d72c1650efc570f93d023c2a8a2a271f8087b26600b8263830/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/cfb0ee759203e2e0067b44098dd189a0800c823be1896e726780bd6b1c285c63/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/06ae4e862f0c1edc86770c0ac4a6e18c021c4dd016b47202120a089dc3a62d71/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/feec4b27e23bbff442004ac99602e803a79e83b0b8532fbe2aba8934179f447c/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/39c20390364abc8ca6af1b6aa1bfd722995cb73cd6006e0e160989163561fa14/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/e5d8db743855a66f22eea4b9ed153ceabb5784d818430585446832e795ab3018/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/172f05c3d5f19c8998f0d27ce9f428bf7b96c1771926e043c2d6d67f60da97cc/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/04a9efa4b2307d57c937782e409318ef55550adf0410a0c917e4e5ab848baa48/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/d4e195ec7669470609db9b63cd1e21e96cb5824712f61acc2e2c53a6641d3f7e/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/47e4e84ffbbfca734417213e99a42f197937500c77e7d3ed84d394b16a3acc6a/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/86f87d81945e3e261f5c62cb207f97e977b3ab24f3b6437a7ffcc423243f8a5f/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/092c6ddecdaef8519c45bc9350c10c5b5c54ae4f2537ca1b3a0c2e8e17ce603f/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/6c66ecd2d7be791b3dce5271ef2f1807d7006e86ef8d1e87e7ca6ad08038300c/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/9a76933b58c4b2c54f69fa7a2b3d4b16659b4a2db1141c9aee394fa26e9f127a/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/908d29ae33ea46f8192e773ddce0182a6c8ec95c1b8f75a5a1846ee31f6498eb/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/544d22089578faae9b414e1208c3ee39c973f213a822fc93b4e96fe56f8e0315/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f65db4c9779d73c4200d90d59bda4cb71bd3fe61ba52a73e856865727b268ac5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f0bbef77771ef52fd41a38dfb51f8457606d667e7a447a2f326a0e50a08e1669/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/711c97f1bec4fba3c358acb8767aa02fb38b91aa0b3c1cd395a3a401726fa36f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/76487a35d9a8ad274dd4289221e528c6b119007196d78de2eef043e1bbe48926/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f151fb3d122e5418970f09c2319fcf65a6af6f6d2be195770f855da3927fcebe/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9c7d529e85e28b1e95297782c17d78c34bd93fec97c3afc0447299f0698b5af1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/14d30b2c49aed1305aabbd9e7a8e4add061987f35a315f2c2288dfb9633fa195/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a3acee9b707af03c479951a0003e60152fc66465428d66d8320dbc83948475b6/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/01c1ce5881662b1a63269b932cd8b81088606c6a33c687ae93c7648c2ae50587/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 20:03:12.393669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.393098 5633 manager.go:217] Machine: {Timestamp:2026-11-01 20:03:12.392408077 +0000 UTC m=+0.066667323 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100701 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff91cd15200a8acd4b0cc08e09498 SystemUUID:ec2ff91c-d152-00a8-acd4-b0cc08e09498 BootID:4237194a-4dc3-4052-be14-7bbb92af4feb Filesystems:[{Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~projected/kube-api-access-rv69v DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~projected/kube-api-access-5tpt9 DeviceMajor:0 DeviceMinor:325 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:367 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0d3a8b55cb40dfd470791821223c8cbbd9f5f27ffc5ab3e2feb8d5aa80e8396/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:323 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:330 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67304fd16b169c449dc4c31ef68631157af0ed6d0956b91f961f0ca09993cc26/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:289 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:324 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db9c2d78f93b4e895f0749ac63cfecd5081b4a2d2387199c42117ca83b46853d/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4c93e2c40fb25c207514420be7c29cb149d1b07952fdd6d50bbb2478a9b0831/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e86d2ca4-1f79-454b-bd82-e86db21a1abc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:401 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7725c3b6cf4533801c588c89317e48d70b3a0850879a957c0794e5146418e84/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a144e458a6f1fdf6107ebae05d781f99e384e1f9282c814aed8bcf4c359c046/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:290 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4561dd90-478e-46b0-805e-b2ead5252a83/volumes/kubernetes.io~projected/kube-api-access-hgm5b DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volumes/kubernetes.io~projected/kube-api-access-rlhh8 DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2a333f0c31ea725139b310d3b8ca0a0833f522d72ee683aa320c99d9b213699/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c049b5fae3b06e7b163dcb50beff63b937223e0c301b4b9421c7b8664d65d7b/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~projected/kube-api-access-qfggb DeviceMajor:0 DeviceMinor:291 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/853d6d5f-e795-45b9-b633-48f2f0051ffc/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:319 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:290 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:394 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/460d8bce-a4b3-4a43-b5cb-f7ff32651690/volumes/kubernetes.io~projected/kube-api-access-647zs DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5464920b79c38dbf826a9b3bc04dc33a62856daa653a58683209e26c50864fa5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~projected/kube-api-access-7shpz DeviceMajor:0 DeviceMinor:277 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d40959f94916497c751f22e6b9995f21018fe0fda22912ebf7952e5c43ae4433/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64bb6b442c763faca4d2f22c0e5cb351188f635cea7161488c120b63ac06bad2/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d5db8c010ce1bc69b6ac71fdf7325bc1072b77aba4a1865db05280e12a3717a/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:397 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:399 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5/volumes/kubernetes.io~projected/kube-api-access-mlt72 DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ac77269f540a824c3bb9ac53fe976f5a6ce8f4178f75492ff12ae62b9ba3d35/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f7737ff-1496-4eed-9bfb-b3e6ea260132/volumes/kubernetes.io~projected/kube-api-access-2rkgr DeviceMajor:0 DeviceMinor:375 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd/userdata/shm DeviceMajor:0 DeviceMinor:65 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/615cde627b6896c432dd15a2527790a8550808513d6fb140f101c7cae202ce09/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~projected/kube-api-access-cjlgj DeviceMajor:0 DeviceMinor:243 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~projected/kube-api-access-sdphh DeviceMajor:0 DeviceMinor:334 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4561dd90-478e-46b0-805e-b2ead5252a83/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:372 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93e86161-3b78-4cd6-960d-70755fcc7b76/volumes/kubernetes.io~projected/kube-api-access-jm4zh DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:331 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc4fc9851fd62b181e86b1bef2ec07dea07783ffe2e17ab12fcc0169beb8cc6e/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f0851759-372d-4706-8bf7-ee6a493872f4/volumes/kubernetes.io~projected/kube-api-access-rqtwv DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:401 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:401 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:284 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33094852f688fab1e75893cf2edfa817e212697837b2fe3d1511f8c2c42fea8e/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e86d2ca4-1f79-454b-bd82-e86db21a1abc/volumes/kubernetes.io~projected/kube-api-access-4wmnv DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:285 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:393 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:398 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fedaa5297a99a3c3a39916c21582943e803408a4355d5943634acee92b277de/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b05d35e-918b-4b8c-9643-78f8dff0b109/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:388 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/85aede5a-5dd5-4665-a7c8-f8efa9eea428/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8658d7e408b8697cf601f810e2d9ca3c27e1f0eb5e2d88020910a2d9b6f8fe69/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:401 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/d979ea87-f591-49d1-a326-bb58c0af72aa/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:400 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef/volumes/kubernetes.io~projected/kube-api-access-ljcc7 DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3a5a4a2b-455d-42ac-8064-cb987052b764/volumes/kubernetes.io~projected/kube-api-access-frr7p DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:282 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b1bf24a-ab20-442b-b54a-b21dd4554ce2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5931bef0-3ad6-4758-82c5-352add308c77/volumes/kubernetes.io~projected/kube-api-access-tg7pl DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc2fb59201257eefec5a5fbbe040c89fc1e0cb7eaad7ad6480f8b3b1395aacb7/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f2c85daa-8760-4950-8cb2-e07c2aa5292e/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:395 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:392 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:396 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4931755e-e9dc-4a2f-aab7-5d64dacb361e/volumes/kubernetes.io~projected/kube-api-access-np598 DeviceMajor:0 DeviceMinor:270 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d11ea05-bc33-4bbc-bcc3-942295c96058/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:332 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes/kubernetes.io~projected/kube-api-access-gjxds DeviceMajor:0 DeviceMinor:402 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0db527d3-b994-4c15-9a6e-3c01ea4b092e/volumes/kubernetes.io~projected/kube-api-access-s5t6p DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93e86161-3b78-4cd6-960d-70755fcc7b76/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f0851759-372d-4706-8bf7-ee6a493872f4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1a144e458a6f1fd MacAddress:c2:67:60:08:10:27 Speed:10000 Mtu:8901} {Name:2d5db8c010ce1bc MacAddress:b2:fc:f3:f1:76:dd Speed:10000 Mtu:8901} {Name:33094852f688fab MacAddress:0e:ea:66:4c:11:b3 Speed:10000 Mtu:8901} {Name:64bb6b442c763fa MacAddress:aa:aa:9e:7b:10:a3 Speed:10000 Mtu:8901} {Name:67304fd16b169c4 MacAddress:3e:88:f3:08:1f:a6 Speed:10000 Mtu:8901} {Name:6d86f56c568c4d7 MacAddress:22:67:f7:c8:0f:06 Speed:10000 Mtu:8901} {Name:7769919e3bf3566 MacAddress:26:25:86:92:44:20 Speed:10000 Mtu:8901} {Name:8658d7e408b8697 MacAddress:0a:f2:a1:61:d9:61 Speed:10000 Mtu:8901} {Name:8c049b5fae3b06e MacAddress:5e:46:5b:b6:f8:14 Speed:10000 Mtu:8901} {Name:bc4fc9851fd62b1 MacAddress:3e:10:34:76:52:5f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:39:3a:64:78:7b Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:11:f1:ff:f7:9d Speed:0 Mtu:8901} {Name:dc2fb59201257ee MacAddress:c6:88:b3:fd:a5:a7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:39:3a:64:78:7b Speed:0 Mtu:9001} {Name:f0d3a8b55cb40df MacAddress:6a:ee:a3:aa:3d:6c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:06:5c:ae:75:77:d5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:b5:9b:41:7e:fb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:03:12.393669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.393674 5633 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:03:12.394013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.393740 5633 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:03:12.394104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394075 5633 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:03:12.394286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394106 5633 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-78.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:03:12.394355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394290 5633 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:03:12.394355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394298 5633 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:03:12.394355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394310 5633 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:03:12.394355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394324 5633 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:03:12.394466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394454 5633 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:03:12.394641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394630 5633 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:03:12.394758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394750 5633 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:03:12.394795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394787 5633 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:03:12.394824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394798 5633 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:03:12.394824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394809 5633 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:03:12.394824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394818 5633 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:03:12.394908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.394826 5633 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:03:12.395680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.395666 5633 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:03:12.395680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.395679 5633 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:03:12.396265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.396242 5633 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:03:12.413296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413268 5633 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:03:12.413296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413288 5633 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:03:12.413474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413404 5633 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 20:03:12.413624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413612 5633 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:03:12.413664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413642 5633 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:03:12.413762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413751 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:03:12.413793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413772 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:03:12.413793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413781 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:03:12.413793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413789 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413795 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413801 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413806 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413812 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413819 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413826 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413842 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413851 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:03:12.413882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413879 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:03:12.414145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.413887 5633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:03:12.418128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.418102 5633 server.go:1263] "Started kubelet" Nov 01 20:03:12.418660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.418622 5633 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:03:12.418685 ip-10-0-128-78 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:03:12.420790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.420143 5633 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:03:12.420790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.420261 5633 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:03:12.420790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.420618 5633 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:03:12.421016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.420805 5633 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:03:12.424192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.423749 5633 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:03:12.424192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.423771 5633 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:03:12.424192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.423828 5633 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:03:12.424192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.423840 5633 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:03:12.424192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.423950 5633 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:03:12.424192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.423966 5633 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:57:01 +0000 UTC" deadline="2028-07-17 06:55:23.143016066 +0000 UTC" Nov 01 20:03:12.424192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.424002 5633 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14962h52m10.71901781s" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426408 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426463 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0851759-372d-4706-8bf7-ee6a493872f4" volumeName="kubernetes.io/configmap/f0851759-372d-4706-8bf7-ee6a493872f4-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426494 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0851759-372d-4706-8bf7-ee6a493872f4" volumeName="kubernetes.io/secret/f0851759-372d-4706-8bf7-ee6a493872f4-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426509 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c85daa-8760-4950-8cb2-e07c2aa5292e" volumeName="kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426525 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426540 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426570 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426584 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85aede5a-5dd5-4665-a7c8-f8efa9eea428" volumeName="kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-tuned" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426604 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426617 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426632 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b1bf24a-ab20-442b-b54a-b21dd4554ce2" volumeName="kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-script-lib" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426651 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" volumeName="kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426664 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426684 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b05d35e-918b-4b8c-9643-78f8dff0b109" volumeName="kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426699 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" volumeName="kubernetes.io/projected/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-kube-api-access-mlt72" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426710 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426724 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4931755e-e9dc-4a2f-aab7-5d64dacb361e" volumeName="kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426738 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426751 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426766 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426781 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b05d35e-918b-4b8c-9643-78f8dff0b109" volumeName="kubernetes.io/configmap/8b05d35e-918b-4b8c-9643-78f8dff0b109-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426803 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d979ea87-f591-49d1-a326-bb58c0af72aa" volumeName="kubernetes.io/secret/d979ea87-f591-49d1-a326-bb58c0af72aa-agent-certs" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426814 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853d6d5f-e795-45b9-b633-48f2f0051ffc" volumeName="kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-client-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426825 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426839 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853d6d5f-e795-45b9-b633-48f2f0051ffc" volumeName="kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426853 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0851759-372d-4706-8bf7-ee6a493872f4" volumeName="kubernetes.io/empty-dir/f0851759-372d-4706-8bf7-ee6a493872f4-data-volume" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426871 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0851759-372d-4706-8bf7-ee6a493872f4" volumeName="kubernetes.io/projected/f0851759-372d-4706-8bf7-ee6a493872f4-kube-api-access-rqtwv" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426884 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426905 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426918 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a5a4a2b-455d-42ac-8064-cb987052b764" volumeName="kubernetes.io/projected/3a5a4a2b-455d-42ac-8064-cb987052b764-kube-api-access-frr7p" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426930 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4931755e-e9dc-4a2f-aab7-5d64dacb361e" volumeName="kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426943 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426958 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b05d35e-918b-4b8c-9643-78f8dff0b109" volumeName="kubernetes.io/projected/8b05d35e-918b-4b8c-9643-78f8dff0b109-kube-api-access-cjlgj" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426974 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b1bf24a-ab20-442b-b54a-b21dd4554ce2" volumeName="kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-env-overrides" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426985 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b1bf24a-ab20-442b-b54a-b21dd4554ce2" volumeName="kubernetes.io/secret/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.426998 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427012 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427036 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-client-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427047 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5931bef0-3ad6-4758-82c5-352add308c77" volumeName="kubernetes.io/projected/5931bef0-3ad6-4758-82c5-352add308c77-kube-api-access-tg7pl" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427057 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e86d2ca4-1f79-454b-bd82-e86db21a1abc" volumeName="kubernetes.io/secret/e86d2ca4-1f79-454b-bd82-e86db21a1abc-cert" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427070 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c85daa-8760-4950-8cb2-e07c2aa5292e" volumeName="kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427082 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef" volumeName="kubernetes.io/empty-dir/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-tmp-dir" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427095 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427108 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c85daa-8760-4950-8cb2-e07c2aa5292e" volumeName="kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427120 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db527d3-b994-4c15-9a6e-3c01ea4b092e" volumeName="kubernetes.io/configmap/0db527d3-b994-4c15-9a6e-3c01ea4b092e-iptables-alerter-script" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427132 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0db527d3-b994-4c15-9a6e-3c01ea4b092e" volumeName="kubernetes.io/projected/0db527d3-b994-4c15-9a6e-3c01ea4b092e-kube-api-access-s5t6p" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427143 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427195 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4931755e-e9dc-4a2f-aab7-5d64dacb361e" volumeName="kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427213 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/projected/4d11ea05-bc33-4bbc-bcc3-942295c96058-kube-api-access-sdphh" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427248 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427260 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e86161-3b78-4cd6-960d-70755fcc7b76" volumeName="kubernetes.io/secret/93e86161-3b78-4cd6-960d-70755fcc7b76-metrics-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427271 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e86d2ca4-1f79-454b-bd82-e86db21a1abc" volumeName="kubernetes.io/projected/e86d2ca4-1f79-454b-bd82-e86db21a1abc-kube-api-access-4wmnv" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427300 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4561dd90-478e-46b0-805e-b2ead5252a83" volumeName="kubernetes.io/projected/4561dd90-478e-46b0-805e-b2ead5252a83-kube-api-access-hgm5b" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427315 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="460d8bce-a4b3-4a43-b5cb-f7ff32651690" volumeName="kubernetes.io/projected/460d8bce-a4b3-4a43-b5cb-f7ff32651690-kube-api-access-647zs" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427358 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427376 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7737ff-1496-4eed-9bfb-b3e6ea260132" volumeName="kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427398 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853d6d5f-e795-45b9-b633-48f2f0051ffc" volumeName="kubernetes.io/empty-dir/853d6d5f-e795-45b9-b633-48f2f0051ffc-audit-log" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427409 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c85daa-8760-4950-8cb2-e07c2aa5292e" volumeName="kubernetes.io/projected/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-api-access-7shpz" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427421 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a5a4a2b-455d-42ac-8064-cb987052b764" volumeName="kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-daemon-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427434 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="460d8bce-a4b3-4a43-b5cb-f7ff32651690" volumeName="kubernetes.io/configmap/460d8bce-a4b3-4a43-b5cb-f7ff32651690-serviceca" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427446 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e86161-3b78-4cd6-960d-70755fcc7b76" volumeName="kubernetes.io/projected/93e86161-3b78-4cd6-960d-70755fcc7b76-kube-api-access-jm4zh" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427460 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b1bf24a-ab20-442b-b54a-b21dd4554ce2" volumeName="kubernetes.io/projected/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-kube-api-access-rlhh8" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427473 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427481 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427488 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427497 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" volumeName="kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-binary-copy" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427505 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c85daa-8760-4950-8cb2-e07c2aa5292e" volumeName="kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427513 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c85daa-8760-4950-8cb2-e07c2aa5292e" volumeName="kubernetes.io/empty-dir/f2c85daa-8760-4950-8cb2-e07c2aa5292e-volume-directive-shadow" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.427057 5633 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427527 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427537 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427549 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427561 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427573 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a5a4a2b-455d-42ac-8064-cb987052b764" volumeName="kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-cni-binary-copy" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427583 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4931755e-e9dc-4a2f-aab7-5d64dacb361e" volumeName="kubernetes.io/projected/4931755e-e9dc-4a2f-aab7-5d64dacb361e-kube-api-access-np598" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427596 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee" volumeName="kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427609 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427621 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef" volumeName="kubernetes.io/projected/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-kube-api-access-ljcc7" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427633 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4931755e-e9dc-4a2f-aab7-5d64dacb361e" volumeName="kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:12.427579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427645 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-federate-client-tls" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427657 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427668 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e86161-3b78-4cd6-960d-70755fcc7b76" volumeName="kubernetes.io/configmap/93e86161-3b78-4cd6-960d-70755fcc7b76-config-volume" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427689 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e86161-3b78-4cd6-960d-70755fcc7b76" volumeName="kubernetes.io/empty-dir/93e86161-3b78-4cd6-960d-70755fcc7b76-tmp-dir" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427702 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b1bf24a-ab20-442b-b54a-b21dd4554ce2" volumeName="kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-config" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427715 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" volumeName="kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427732 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427743 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427753 5633 factory.go:55] Registering systemd factory Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427769 5633 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427756 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427828 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4931755e-e9dc-4a2f-aab7-5d64dacb361e" volumeName="kubernetes.io/empty-dir/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-textfile" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427852 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427861 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853d6d5f-e795-45b9-b633-48f2f0051ffc" volumeName="kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-metrics-server-audit-profiles" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427871 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853d6d5f-e795-45b9-b633-48f2f0051ffc" volumeName="kubernetes.io/projected/853d6d5f-e795-45b9-b633-48f2f0051ffc-kube-api-access-5tpt9" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427881 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853d6d5f-e795-45b9-b633-48f2f0051ffc" volumeName="kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-tls" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427889 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fab5652-b1da-42d8-9c42-1e32ad4fc054" volumeName="kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427902 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4561dd90-478e-46b0-805e-b2ead5252a83" volumeName="kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427912 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853d6d5f-e795-45b9-b633-48f2f0051ffc" volumeName="kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427920 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85aede5a-5dd5-4665-a7c8-f8efa9eea428" volumeName="kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-tmp" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427937 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427947 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" volumeName="kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427958 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b05d35e-918b-4b8c-9643-78f8dff0b109" volumeName="kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-tls" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427976 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d979ea87-f591-49d1-a326-bb58c0af72aa" volumeName="kubernetes.io/configmap/d979ea87-f591-49d1-a326-bb58c0af72aa-konnectivity-ca" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427984 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d11ea05-bc33-4bbc-bcc3-942295c96058" volumeName="kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427991 5633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85aede5a-5dd5-4665-a7c8-f8efa9eea428" volumeName="kubernetes.io/projected/85aede5a-5dd5-4665-a7c8-f8efa9eea428-kube-api-access-rv69v" seLinuxMountContext="" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.427999 5633 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.428006 5633 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.428454 5633 factory.go:153] Registering CRI-O factory Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.428472 5633 factory.go:223] Registration of the crio container factory successfully Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.428812 5633 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.428839 5633 factory.go:103] Registering Raw factory Nov 01 20:03:12.431043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.428852 5633 manager.go:1199] Started watching for new ooms in manager Nov 01 20:03:12.433892 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.431350 5633 manager.go:319] Starting recovery of all containers Nov 01 20:03:12.442447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.442406 5633 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:03:12.458985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.458952 5633 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:03:12.458985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.458980 5633 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:03:12.459207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.459001 5633 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:03:12.459207 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.459043 5633 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 20:03:12.474521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.474483 5633 generic.go:356] "Generic (PLEG): container finished" podID="4931755e-e9dc-4a2f-aab7-5d64dacb361e" containerID="d0d924d27e1d42f3b666e76386101cf4afa1a8d207c244b9a4abc647bc099fff" exitCode=0 Nov 01 20:03:12.481948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.481910 5633 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="53e47a8b583acdd21eed7965d05e47dcaf5c830bce5e06f4832847093ae8d972" exitCode=0 Nov 01 20:03:12.481948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.481945 5633 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="00a75bc40c67340fe2f340038461b1015c5369892f9a778238c9f313610a7904" exitCode=0 Nov 01 20:03:12.481948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.481955 5633 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="1bce52648b308eb5910b7a3bc828d447dfbd34ea9b15c32a538e693366e9cfd5" exitCode=0 Nov 01 20:03:12.482186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.481968 5633 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="6b6c634515ec1cafe2144cd0e1d3d37327d98be3c3b5fc044526616854ab4f92" exitCode=0 Nov 01 20:03:12.482186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.481976 5633 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="6980897864b690c2a7d7726e99493c1ec5177d36247fc5aceab84d126c53488a" exitCode=0 Nov 01 20:03:12.482186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.481985 5633 generic.go:356] "Generic (PLEG): container finished" podID="d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5" containerID="7e770d18e2bbca5656dae7eb51ac3c40c6890abb51a6fd3911fdec28e858afed" exitCode=0 Nov 01 20:03:12.502756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.502723 5633 generic.go:356] "Generic (PLEG): container finished" podID="9da5f29a924a82bf1fb8d408f5c8f1a7" containerID="2b3f4815ff02fce3d44fe5d03eb829d10338fc08e576b58fa927d8f53a34a84b" exitCode=0 Nov 01 20:03:12.505694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.505655 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" exitCode=0 Nov 01 20:03:12.511184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.511152 5633 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="712284bdce2a4f3561ca2a1b4288fc6a91da6ef981e35e8968dc028b7151ed58" exitCode=0 Nov 01 20:03:12.527638 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.527566 5633 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" Nov 01 20:03:12.548858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.548833 5633 manager.go:324] Recovery completed Nov 01 20:03:12.552308 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.552273 5633 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:12.552308 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.552279 5633 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:12.559523 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.559500 5633 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 20:03:12.570472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.570444 5633 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:12.572675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.572646 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:03:12.572789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.572688 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:03:12.572789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.572699 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:03:12.575752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575732 5633 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:03:12.575752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575747 5633 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:03:12.575900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575752 5633 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:12.575900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575765 5633 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:03:12.575900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575889 5633 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 20:03:12.575996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575899 5633 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 20:03:12.575996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575915 5633 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 20:03:12.575996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575920 5633 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 20:03:12.575996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575926 5633 policy_none.go:50] "Start" Nov 01 20:03:12.575996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575935 5633 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:03:12.575996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.575944 5633 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:03:12.576180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.576067 5633 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 20:03:12.576180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.576073 5633 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 20:03:12.576180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.576079 5633 policy_none.go:44] "Start" Nov 01 20:03:12.576689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.576666 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:03:12.576774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.576703 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:03:12.576774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.576718 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:03:12.580989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.580960 5633 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:03:12.583691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.583675 5633 manager.go:341] "Starting Device Plugin manager" Nov 01 20:03:12.584156 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.583701 5633 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:03:12.584196 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584158 5633 server.go:87] "Starting device plugin registration server" Nov 01 20:03:12.584457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584446 5633 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:03:12.584492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584459 5633 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:03:12.584560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584544 5633 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:03:12.584669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584631 5633 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:03:12.584766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584682 5633 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:03:12.584766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584689 5633 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:03:12.584873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.584754 5633 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:03:12.584663541Z","UUID":"d02feef0-135e-4167-a274-1867833763da","Handler":null,"Name":"","Endpoint":""} Nov 01 20:03:12.586237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.586219 5633 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:03:12.586381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.586247 5633 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:03:12.586479 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.586459 5633 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:03:12.586537 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.586510 5633 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-78.us-west-1.compute.internal\" not found" Nov 01 20:03:12.602116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.602095 5633 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:12.602116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.602115 5633 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 20:03:12.684584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.684555 5633 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:12.685667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.685648 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:03:12.685818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.685687 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:03:12.685818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.685703 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:03:12.685818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.685766 5633 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.699526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.699502 5633 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:12.760443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.760375 5633 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal"] Nov 01 20:03:12.761082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.761036 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"c04d0eff16222e7baef223e711863fea","Type":"ContainerStarted","Data":"12481e4a0c1285da036c811dcbe36efa6ee4dfd7321c5ae15d10da7382e9d2ad"} Nov 01 20:03:12.761188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.761090 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"c04d0eff16222e7baef223e711863fea","Type":"ContainerStarted","Data":"cdc905bef362386cc4a5134636981461e5e383e18b44c1be38fceac3d30bd225"} Nov 01 20:03:12.761188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.761109 5633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.761188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.761145 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"9da5f29a924a82bf1fb8d408f5c8f1a7","Type":"ContainerStarted","Data":"276c62c3b4498e2d61ca2335e4f09ab67b341ff0cb18e93cced5f44f0ffff5e2"} Nov 01 20:03:12.761188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.761169 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"9da5f29a924a82bf1fb8d408f5c8f1a7","Type":"ContainerDied","Data":"2b3f4815ff02fce3d44fe5d03eb829d10338fc08e576b58fa927d8f53a34a84b"} Nov 01 20:03:12.761188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.761185 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" event={"ID":"9da5f29a924a82bf1fb8d408f5c8f1a7","Type":"ContainerStarted","Data":"94d961f53b49f5f951d84b425f6da594214b6492fbff8295c885c66fcaa706cd"} Nov 01 20:03:12.761390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.761346 5633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.805938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.805867 5633 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.805938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.805936 5633 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.822497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.822459 5633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:12.822646 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.822509 5633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.822715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.822702 5633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:12.822767 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:12.822735 5633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.904606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.904561 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.904778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.904615 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c04d0eff16222e7baef223e711863fea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"c04d0eff16222e7baef223e711863fea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:12.904778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:12.904694 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.005439 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.005405 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.005439 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.005442 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.005650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.005461 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c04d0eff16222e7baef223e711863fea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"c04d0eff16222e7baef223e711863fea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.005650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.005526 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c04d0eff16222e7baef223e711863fea-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"c04d0eff16222e7baef223e711863fea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.005650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.005530 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.005650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.005593 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da5f29a924a82bf1fb8d408f5c8f1a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" (UID: \"9da5f29a924a82bf1fb8d408f5c8f1a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.395197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.395162 5633 apiserver.go:52] "Watching apiserver" Nov 01 20:03:13.517011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.516935 5633 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:03:13.517614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.517592 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d569ccc7c-67pl4","openshift-dns/node-resolver-ct9f5","openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-qm6s6","openshift-network-operator/iptables-alerter-2kzz5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal","openshift-monitoring/telemeter-client-9f57576c8-qm8gx","kube-system/konnectivity-agent-8rbsz","openshift-cluster-node-tuning-operator/tuned-h8n5g","openshift-insights/insights-runtime-extractor-jwwcz","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-2kxmx","kube-system/global-pull-secret-syncer-7lwmf","openshift-dns/dns-default-lthd5","openshift-ovn-kubernetes/ovnkube-node-f9xbg","kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-ks9th","openshift-monitoring/node-exporter-9hnrf","openshift-multus/multus-additional-cni-plugins-7bjpt","openshift-image-registry/node-ca-zd4r8","openshift-ingress-canary/ingress-canary-2v4wp","openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr"] Nov 01 20:03:13.532361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.531919 5633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.532361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.532099 5633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.572976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.572940 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.572976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.572962 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.572976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.572940 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:03:13.572976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.572940 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.572976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.572948 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.572976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.572940 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:03:13.572976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573018 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.573465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573024 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.573465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573053 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.573465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573068 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.573465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573078 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:03:13.573465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573081 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.573465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573129 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:03:13.573465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573147 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.574051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573673 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:03:13.574051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573758 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\"" Nov 01 20:03:13.574051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573766 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:03:13.574051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573871 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\"" Nov 01 20:03:13.574051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.573915 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:03:13.574314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574118 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\"" Nov 01 20:03:13.574314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574128 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\"" Nov 01 20:03:13.574314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574181 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" Nov 01 20:03:13.574314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574201 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" Nov 01 20:03:13.574314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574209 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:03:13.574314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574265 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\"" Nov 01 20:03:13.574599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574364 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\"" Nov 01 20:03:13.574599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574377 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:03:13.574599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574424 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\"" Nov 01 20:03:13.574599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574468 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" Nov 01 20:03:13.574599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574605 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:03:13.574817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574664 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.574952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.574936 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.575229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575212 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:03:13.575397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575243 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.575510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575419 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:03:13.575510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575454 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:03:13.575510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575503 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.575640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575510 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\"" Nov 01 20:03:13.575640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575544 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.575640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575590 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.575640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575606 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.575794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575683 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\"" Nov 01 20:03:13.575794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575689 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:03:13.575794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575687 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:03:13.575794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575702 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\"" Nov 01 20:03:13.575794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575698 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\"" Nov 01 20:03:13.575794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575720 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.575979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575802 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:03:13.575979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.575860 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:03:13.610060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610027 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-hosts-file\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:03:13.610060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610063 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-modprobe-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.610263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610084 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-host\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.610263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610172 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.610263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610199 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d979ea87-f591-49d1-a326-bb58c0af72aa-konnectivity-ca\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:03:13.610263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610223 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhh8\" (UniqueName: \"kubernetes.io/projected/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-kube-api-access-rlhh8\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.610263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610256 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460d8bce-a4b3-4a43-b5cb-f7ff32651690-host\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:03:13.610548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610284 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-tmp-dir\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:03:13.610548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610309 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4zh\" (UniqueName: \"kubernetes.io/projected/93e86161-3b78-4cd6-960d-70755fcc7b76-kube-api-access-jm4zh\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.610548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610374 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.610548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610400 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-node-log\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.610548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610493 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.610548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610496 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d979ea87-f591-49d1-a326-bb58c0af72aa-konnectivity-ca\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:03:13.610548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610528 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-tmp-dir\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610494 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93e86161-3b78-4cd6-960d-70755fcc7b76-metrics-tls\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610601 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-tmp\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610627 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-kubelet\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610645 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-sys\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610668 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db527d3-b994-4c15-9a6e-3c01ea4b092e-host-slash\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610683 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-tmp\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610689 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-netns\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.610790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610767 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-bin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610807 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610826 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-run\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610841 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-tuned\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610859 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0851759-372d-4706-8bf7-ee6a493872f4-crio-socket\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610887 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-socket-dir-parent\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610913 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7pl\" (UniqueName: \"kubernetes.io/projected/5931bef0-3ad6-4758-82c5-352add308c77-kube-api-access-tg7pl\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610947 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-tuned\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610966 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-slash\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.610993 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-var-lib-kubelet\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611034 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-kubelet\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611070 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frr7p\" (UniqueName: \"kubernetes.io/projected/3a5a4a2b-455d-42ac-8064-cb987052b764-kube-api-access-frr7p\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.611124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611069 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4561dd90-478e-46b0-805e-b2ead5252a83-metrics-certs\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:03:13.611469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611195 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647zs\" (UniqueName: \"kubernetes.io/projected/460d8bce-a4b3-4a43-b5cb-f7ff32651690-kube-api-access-647zs\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:03:13.611469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611225 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-kubernetes\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611248 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611275 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0851759-372d-4706-8bf7-ee6a493872f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.611469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611302 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e86d2ca4-1f79-454b-bd82-e86db21a1abc-cert\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:03:13.611469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611356 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.611645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611468 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-device-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.611645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611517 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0851759-372d-4706-8bf7-ee6a493872f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.611645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611523 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.611645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611554 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-var-lib-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.611645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611571 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-etc-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.611645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611615 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e86161-3b78-4cd6-960d-70755fcc7b76-tmp-dir\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.611645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611642 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-conf\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611659 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-lib-modules\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.611842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611750 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0851759-372d-4706-8bf7-ee6a493872f4-data-volume\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.611842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611749 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e86161-3b78-4cd6-960d-70755fcc7b76-tmp-dir\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.611842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611773 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-config\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.611842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611816 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460d8bce-a4b3-4a43-b5cb-f7ff32651690-serviceca\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:03:13.611842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611827 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0851759-372d-4706-8bf7-ee6a493872f4-data-volume\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.612001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611841 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-systemd\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.612001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.611916 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5t6p\" (UniqueName: \"kubernetes.io/projected/0db527d3-b994-4c15-9a6e-3c01ea4b092e-kube-api-access-s5t6p\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:03:13.612054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612003 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-hostroot\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.612054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612009 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-config\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612106 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcc7\" (UniqueName: \"kubernetes.io/projected/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-kube-api-access-ljcc7\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:03:13.612194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612178 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtwv\" (UniqueName: \"kubernetes.io/projected/f0851759-372d-4706-8bf7-ee6a493872f4-kube-api-access-rqtwv\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.612251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612234 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-k8s-cni-cncf-io\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.612300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612284 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-multus\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.612365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612345 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-etc-kubernetes\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.612428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612380 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlt72\" (UniqueName: \"kubernetes.io/projected/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-kube-api-access-mlt72\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.612472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612429 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612559 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460d8bce-a4b3-4a43-b5cb-f7ff32651690-serviceca\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:03:13.612646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612601 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-netd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612621 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612639 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-env-overrides\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612660 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovn-node-metrics-cert\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612691 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysconfig\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.612837 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612790 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-systemd-units\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612837 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612803 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-env-overrides\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612834 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612863 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-ovn\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612889 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-log-socket\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.612994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.612960 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-script-lib\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.613094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613002 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovn-node-metrics-cert\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.613094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613002 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgm5b\" (UniqueName: \"kubernetes.io/projected/4561dd90-478e-46b0-805e-b2ead5252a83-kube-api-access-hgm5b\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:03:13.613094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613047 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-multus-certs\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.613094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613073 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-registration-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.613094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613088 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-system-cni-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.613298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613102 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cnibin\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.613298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613122 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-netns\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.613298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613162 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:13.613298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613190 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-ovnkube-script-lib\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.613298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613288 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-kubelet-config\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.613525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613318 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-cni-binary-copy\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.613525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613349 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d979ea87-f591-49d1-a326-bb58c0af72aa-agent-certs\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:03:13.613525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613388 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93e86161-3b78-4cd6-960d-70755fcc7b76-config-volume\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.613525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613419 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmnv\" (UniqueName: \"kubernetes.io/projected/e86d2ca4-1f79-454b-bd82-e86db21a1abc-kube-api-access-4wmnv\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:03:13.613525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613447 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-cnibin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.613525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613479 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-conf-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.613525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613506 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-socket-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.613780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613549 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.613780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613587 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-sys-fs\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.613780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613616 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d979ea87-f591-49d1-a326-bb58c0af72aa-agent-certs\") pod \"konnectivity-agent-8rbsz\" (UID: \"d979ea87-f591-49d1-a326-bb58c0af72aa\") " pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:03:13.613780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613626 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.613780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613677 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-bin\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.613780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613688 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-cni-binary-copy\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.613780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613765 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db527d3-b994-4c15-9a6e-3c01ea4b092e-iptables-alerter-script\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:03:13.613963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613794 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.613963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613802 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-system-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.613963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613827 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-daemon-config\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.613963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613889 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-systemd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.614077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613977 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-dbus\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.614077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.613995 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.614077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614004 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0db527d3-b994-4c15-9a6e-3c01ea4b092e-iptables-alerter-script\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:03:13.614077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614030 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv69v\" (UniqueName: \"kubernetes.io/projected/85aede5a-5dd5-4665-a7c8-f8efa9eea428-kube-api-access-rv69v\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.614077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614064 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-daemon-config\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.614298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614116 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0851759-372d-4706-8bf7-ee6a493872f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.614298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614224 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.614393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614297 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-original-pull-secret\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.614432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614417 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0851759-372d-4706-8bf7-ee6a493872f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.614464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.614438 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.626089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.626055 5633 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:03:13.629006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.628989 5633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:13.629074 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:13.629067 5633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.631987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.631908 5633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:13.631987 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:13.631974 5633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.us-west-1.compute.internal" Nov 01 20:03:13.671253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.671160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4zh\" (UniqueName: \"kubernetes.io/projected/93e86161-3b78-4cd6-960d-70755fcc7b76-kube-api-access-jm4zh\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.671253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.671214 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7pl\" (UniqueName: \"kubernetes.io/projected/5931bef0-3ad6-4758-82c5-352add308c77-kube-api-access-tg7pl\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.671253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.671223 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhh8\" (UniqueName: \"kubernetes.io/projected/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-kube-api-access-rlhh8\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.671253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.671160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtwv\" (UniqueName: \"kubernetes.io/projected/f0851759-372d-4706-8bf7-ee6a493872f4-kube-api-access-rqtwv\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.672397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.671312 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5t6p\" (UniqueName: \"kubernetes.io/projected/0db527d3-b994-4c15-9a6e-3c01ea4b092e-kube-api-access-s5t6p\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:03:13.672397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.671434 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frr7p\" (UniqueName: \"kubernetes.io/projected/3a5a4a2b-455d-42ac-8064-cb987052b764-kube-api-access-frr7p\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.672397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.671646 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-647zs\" (UniqueName: \"kubernetes.io/projected/460d8bce-a4b3-4a43-b5cb-f7ff32651690-kube-api-access-647zs\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:03:13.672397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.672117 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv69v\" (UniqueName: \"kubernetes.io/projected/85aede5a-5dd5-4665-a7c8-f8efa9eea428-kube-api-access-rv69v\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.672397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.672116 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlt72\" (UniqueName: \"kubernetes.io/projected/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-kube-api-access-mlt72\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.672397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.672183 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgm5b\" (UniqueName: \"kubernetes.io/projected/4561dd90-478e-46b0-805e-b2ead5252a83-kube-api-access-hgm5b\") pod \"network-metrics-daemon-qm6s6\" (UID: \"4561dd90-478e-46b0-805e-b2ead5252a83\") " pod="openshift-multus/network-metrics-daemon-qm6s6" Nov 01 20:03:13.673797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.673775 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkgr\" (UniqueName: \"kubernetes.io/projected/5f7737ff-1496-4eed-9bfb-b3e6ea260132-kube-api-access-2rkgr\") pod \"network-check-target-ks9th\" (UID: \"5f7737ff-1496-4eed-9bfb-b3e6ea260132\") " pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:13.674277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.674256 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 20:03:13.675180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.675154 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 20:03:13.675273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.675205 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcc7\" (UniqueName: \"kubernetes.io/projected/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-kube-api-access-ljcc7\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:03:13.676350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.676311 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 20:03:13.676526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.676503 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ft87h\"" Nov 01 20:03:13.676655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.676636 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 20:03:13.676856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.676840 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 20:03:13.676925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.676866 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 20:03:13.677043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677022 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 20:03:13.677121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677103 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 20:03:13.677230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677211 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 20:03:13.677426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677410 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 20:03:13.677426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677432 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.677558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677498 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2r24h\"" Nov 01 20:03:13.677558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677543 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 20:03:13.677687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677571 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 20:03:13.677687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677647 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 20:03:13.677794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677703 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 20:03:13.677856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677813 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 20:03:13.677918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677885 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 20:03:13.677918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677892 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 20:03:13.678024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.677981 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ljq7\"" Nov 01 20:03:13.678114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678096 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 20:03:13.678242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678222 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6vk7hqa1puoju\"" Nov 01 20:03:13.678309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678249 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 20:03:13.678309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678255 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 20:03:13.678309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678267 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 20:03:13.678518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678463 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 20:03:13.678581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678543 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\"" Nov 01 20:03:13.678690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678675 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:03:13.678773 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678757 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 20:03:13.678773 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678766 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-76lth\"" Nov 01 20:03:13.678874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.678867 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 20:03:13.679091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679073 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sbflc\"" Nov 01 20:03:13.679157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679135 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 20:03:13.679215 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679186 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.679285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679270 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 20:03:13.679347 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679320 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:03:13.679395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679355 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 20:03:13.679531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679511 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:03:13.679605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679532 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c9o03dg9n84k5\"" Nov 01 20:03:13.679605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679598 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 20:03:13.679703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679608 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 20:03:13.679748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679733 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 20:03:13.679894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679872 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:03:13.679953 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679903 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 20:03:13.680052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.680031 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:03:13.680099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.680057 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 20:03:13.680172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.679534 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\"" Nov 01 20:03:13.680397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.680315 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 20:03:13.680397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.680398 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 20:03:13.680569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.680506 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jh6hf\"" Nov 01 20:03:13.680910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.680893 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 20:03:13.680970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.680943 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 20:03:13.684181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.681026 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93e86161-3b78-4cd6-960d-70755fcc7b76-metrics-tls\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.684181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.681072 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 20:03:13.684181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.681079 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 20:03:13.684472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.684431 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e86d2ca4-1f79-454b-bd82-e86db21a1abc-cert\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:03:13.684472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.684431 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93e86161-3b78-4cd6-960d-70755fcc7b76-config-volume\") pod \"dns-default-lthd5\" (UID: \"93e86161-3b78-4cd6-960d-70755fcc7b76\") " pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:13.684888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.684868 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmnv\" (UniqueName: \"kubernetes.io/projected/e86d2ca4-1f79-454b-bd82-e86db21a1abc-kube-api-access-4wmnv\") pod \"ingress-canary-2v4wp\" (UID: \"e86d2ca4-1f79-454b-bd82-e86db21a1abc\") " pod="openshift-ingress-canary/ingress-canary-2v4wp" Nov 01 20:03:13.715559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715505 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-slash\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.715559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715561 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-kubelet\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715582 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlgj\" (UniqueName: \"kubernetes.io/projected/8b05d35e-918b-4b8c-9643-78f8dff0b109-kube-api-access-cjlgj\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715598 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715613 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715627 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715635 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-kubelet\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715635 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-slash\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715645 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-device-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715700 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715725 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715741 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715744 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-etc-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715802 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/4d11ea05-bc33-4bbc-bcc3-942295c96058-kube-api-access-sdphh\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715812 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.715830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715826 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715839 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-device-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715863 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-systemd\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715893 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-tls\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715921 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-metrics-client-ca\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715942 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715945 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715972 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715974 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715999 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-multus\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716005 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.715980 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-etc-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716038 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-multus\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716040 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716028 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716077 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716105 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716130 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716159 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-tls\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716164 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysconfig\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716208 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716215 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysconfig\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716237 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716248 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716263 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-metrics-client-ca\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716289 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-systemd-units\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716302 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-systemd\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716312 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-ovn\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716363 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-multus-certs\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716367 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716397 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-registration-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716237 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-metrics-client-ca\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716425 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/853d6d5f-e795-45b9-b633-48f2f0051ffc-audit-log\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716439 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716443 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716451 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716455 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b05d35e-918b-4b8c-9643-78f8dff0b109-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716479 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-system-cni-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716503 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cnibin\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716507 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-ovn\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716516 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-registration-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716526 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-netns\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716547 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-multus-certs\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716562 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-cnibin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716582 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716587 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716588 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-systemd-units\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716588 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-socket-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716631 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-netns\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716634 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716640 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-cnibin\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716652 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-cnibin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716212 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716665 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-sys-fs\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716682 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-socket-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716684 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716689 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-bin\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716665 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/853d6d5f-e795-45b9-b633-48f2f0051ffc-audit-log\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716695 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716705 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5-system-cni-dir\") pod \"multus-additional-cni-plugins-7bjpt\" (UID: \"d1ad15ca-e56c-4ddf-9bac-a0243c4f85e5\") " pod="openshift-multus/multus-additional-cni-plugins-7bjpt" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716598 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-metrics-client-ca\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716721 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716727 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-bin\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716717 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-sys-fs\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716758 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-serving-certs-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716786 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-systemd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716808 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-dbus\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716835 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716862 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-modprobe-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716867 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-systemd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716869 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716885 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-client-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716903 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-dbus\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716910 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716934 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716944 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-serving-certs-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716956 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716970 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-modprobe-d\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.716983 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717007 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-node-log\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717008 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717058 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717063 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-node-log\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717089 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717113 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717115 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-kubelet\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717130 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-client-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717155 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-kubelet\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717155 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-sys\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717191 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-netns\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717208 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-bin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717223 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717238 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717239 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-netns\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717254 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717265 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-var-lib-cni-bin\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717271 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717206 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-sys\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717292 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717321 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717320 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-run\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717373 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-socket-dir-parent\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717382 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-run\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717399 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-var-lib-kubelet\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717422 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b05d35e-918b-4b8c-9643-78f8dff0b109-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717428 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717435 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-socket-dir-parent\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717444 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717445 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717470 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-var-lib-kubelet\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717473 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717490 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-kubernetes\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717508 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717524 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-wtmp\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717540 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-var-lib-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717555 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-conf\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717570 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-lib-modules\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717576 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717584 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-hostroot\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717602 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b05d35e-918b-4b8c-9643-78f8dff0b109-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717635 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717664 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5931bef0-3ad6-4758-82c5-352add308c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbndv\" (UID: \"5931bef0-3ad6-4758-82c5-352add308c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbndv" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717652 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-kubernetes\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717677 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717692 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-var-lib-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717694 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717707 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-hostroot\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717665 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-lib-modules\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717797 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-etc-sysctl-conf\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717717 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717806 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717819 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-root\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717859 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-k8s-cni-cncf-io\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717874 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-etc-kubernetes\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717895 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717918 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-textfile\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717929 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-host-run-k8s-cni-cncf-io\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717935 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shpz\" (UniqueName: \"kubernetes.io/projected/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-api-access-7shpz\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717969 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-etc-kubernetes\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717967 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717999 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717996 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-netd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718011 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718034 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-cni-netd\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718036 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718014 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.717895 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718097 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-textfile\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718145 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718159 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718158 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718198 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718291 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-log-socket\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718347 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-run-openvswitch\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718315 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718366 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b1bf24a-ab20-442b-b54a-b21dd4554ce2-log-socket\") pod \"ovnkube-node-f9xbg\" (UID: \"9b1bf24a-ab20-442b-b54a-b21dd4554ce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718377 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np598\" (UniqueName: \"kubernetes.io/projected/4931755e-e9dc-4a2f-aab7-5d64dacb361e-kube-api-access-np598\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718406 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-kubelet-config\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718449 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718504 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-tls\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.718417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718512 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee-kubelet-config\") pod \"global-pull-secret-syncer-7lwmf\" (UID: \"7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee\") " pod="kube-system/global-pull-secret-syncer-7lwmf" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718611 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-conf-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718647 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718659 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-conf-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718679 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718689 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfggb\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718798 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-system-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718790 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-tls\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718822 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718838 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718857 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-system-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718867 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjxds\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718867 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2c85daa-8760-4950-8cb2-e07c2aa5292e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718902 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.718929 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719040 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-hosts-file\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719049 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719063 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719068 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-host\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719108 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef-hosts-file\") pod \"node-resolver-ct9f5\" (UID: \"3ac97caa-7b32-44f7-8ac6-01fe4a9db8ef\") " pod="openshift-dns/node-resolver-ct9f5" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719108 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719112 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85aede5a-5dd5-4665-a7c8-f8efa9eea428-host\") pod \"tuned-h8n5g\" (UID: \"85aede5a-5dd5-4665-a7c8-f8efa9eea428\") " pod="openshift-cluster-node-tuning-operator/tuned-h8n5g" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719113 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-metrics-server-audit-profiles\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719217 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719248 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460d8bce-a4b3-4a43-b5cb-f7ff32651690-host\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719277 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719304 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719354 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719379 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/853d6d5f-e795-45b9-b633-48f2f0051ffc-metrics-server-audit-profiles\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719387 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db527d3-b994-4c15-9a6e-3c01ea4b092e-host-slash\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719411 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719436 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719449 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a5a4a2b-455d-42ac-8064-cb987052b764-multus-cni-dir\") pod \"multus-2kxmx\" (UID: \"3a5a4a2b-455d-42ac-8064-cb987052b764\") " pod="openshift-multus/multus-2kxmx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719460 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0851759-372d-4706-8bf7-ee6a493872f4-crio-socket\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719487 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-client-certs\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719512 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpt9\" (UniqueName: \"kubernetes.io/projected/853d6d5f-e795-45b9-b633-48f2f0051ffc-kube-api-access-5tpt9\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719539 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719565 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-sys\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719586 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719590 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-federate-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719615 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2c85daa-8760-4950-8cb2-e07c2aa5292e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719635 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460d8bce-a4b3-4a43-b5cb-f7ff32651690-host\") pod \"node-ca-zd4r8\" (UID: \"460d8bce-a4b3-4a43-b5cb-f7ff32651690\") " pod="openshift-image-registry/node-ca-zd4r8" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719844 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/853d6d5f-e795-45b9-b633-48f2f0051ffc-secret-metrics-server-client-certs\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719853 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.719901 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0851759-372d-4706-8bf7-ee6a493872f4-crio-socket\") pod \"insights-runtime-extractor-jwwcz\" (UID: \"f0851759-372d-4706-8bf7-ee6a493872f4\") " pod="openshift-insights/insights-runtime-extractor-jwwcz" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.720118 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.720173 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0db527d3-b994-4c15-9a6e-3c01ea4b092e-host-slash\") pod \"iptables-alerter-2kzz5\" (UID: \"0db527d3-b994-4c15-9a6e-3c01ea4b092e\") " pod="openshift-network-operator/iptables-alerter-2kzz5" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.720221 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.720316 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.720481 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d11ea05-bc33-4bbc-bcc3-942295c96058-federate-client-tls\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.720545 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.722524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.720662 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2c85daa-8760-4950-8cb2-e07c2aa5292e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.724871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.724821 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 20:03:13.724978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.724896 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 20:03:13.725164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.725148 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 20:03:13.725380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.725361 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 20:03:13.727425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.727400 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.727486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.727425 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.731411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.731305 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d11ea05-bc33-4bbc-bcc3-942295c96058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.732547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.732512 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.772109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.772065 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlgj\" (UniqueName: \"kubernetes.io/projected/8b05d35e-918b-4b8c-9643-78f8dff0b109-kube-api-access-cjlgj\") pod \"prometheus-operator-9cd89b74c-7v6xv\" (UID: \"8b05d35e-918b-4b8c-9643-78f8dff0b109\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7v6xv" Nov 01 20:03:13.772267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.772189 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/4d11ea05-bc33-4bbc-bcc3-942295c96058-kube-api-access-sdphh\") pod \"telemeter-client-9f57576c8-qm8gx\" (UID: \"4d11ea05-bc33-4bbc-bcc3-942295c96058\") " pod="openshift-monitoring/telemeter-client-9f57576c8-qm8gx" Nov 01 20:03:13.777273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.777236 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shpz\" (UniqueName: \"kubernetes.io/projected/f2c85daa-8760-4950-8cb2-e07c2aa5292e-kube-api-access-7shpz\") pod \"kube-state-metrics-5dc8d7d89b-586nr\" (UID: \"f2c85daa-8760-4950-8cb2-e07c2aa5292e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-586nr" Nov 01 20:03:13.777601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.777580 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfggb\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\") pod \"alertmanager-main-0\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:13.778217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.778195 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np598\" (UniqueName: \"kubernetes.io/projected/4931755e-e9dc-4a2f-aab7-5d64dacb361e-kube-api-access-np598\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.778405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.778390 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpt9\" (UniqueName: \"kubernetes.io/projected/853d6d5f-e795-45b9-b633-48f2f0051ffc-kube-api-access-5tpt9\") pod \"metrics-server-d569ccc7c-67pl4\" (UID: \"853d6d5f-e795-45b9-b633-48f2f0051ffc\") " pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:13.778990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.778973 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjxds\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\") pod \"prometheus-k8s-0\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:13.814911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.814856 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:13.820103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.820070 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-wtmp\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.820291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.820117 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-root\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.820291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.820167 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-sys\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.820291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.820251 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-sys\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.820483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.820312 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-node-exporter-wtmp\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.820483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.820378 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4931755e-e9dc-4a2f-aab7-5d64dacb361e-root\") pod \"node-exporter-9hnrf\" (UID: \"4931755e-e9dc-4a2f-aab7-5d64dacb361e\") " pod="openshift-monitoring/node-exporter-9hnrf" Nov 01 20:03:13.833883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:13.833861 5633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:03:14.009807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:14.009776 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:14.507202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:14.507169 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:14.691542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:14.691504 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lthd5" Nov 01 20:03:14.849734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:14.849666 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:14.928736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:14.928703 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:14.950650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:14.950624 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:15.540280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:15.540251 5633 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:03:16.158182 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:16.158149 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:03:17.167517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.167432 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:17.314479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.314432 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:17.430811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.430726 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:17.552767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.552726 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerStarted","Data":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} Nov 01 20:03:17.552971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.552823 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerStarted","Data":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} Nov 01 20:03:17.552971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.552873 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerStarted","Data":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} Nov 01 20:03:17.552971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.552906 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerStarted","Data":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} Nov 01 20:03:17.554098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.554070 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:17.671732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:17.671700 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:18.111641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:18.111599 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:18.287567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:18.287528 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:19.309154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:19.309108 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:19.515717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:19.515685 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:19.668886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:19.668817 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f9xbg" Nov 01 20:03:23.624587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:23.624492 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:25.141176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:25.141134 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:25.757398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:25.757354 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:25.927603 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:25.927564 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:27.077763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:27.077726 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:27.946622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:27.946584 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:28.065870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:28.065834 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:28.811031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:28.810999 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8rbsz" Nov 01 20:03:28.991298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:28.991266 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bdbf44876-vjp4b"] Nov 01 20:03:28.993864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:28.993846 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.046710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.046670 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-tmp\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.046710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.046713 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfxt\" (UniqueName: \"kubernetes.io/projected/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-kube-api-access-xrfxt\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.046917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.046811 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-service-ca\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.046917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.046839 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-trusted-ca-bundle\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.046983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.046927 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-oauth-serving-cert\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.046983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.046952 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-serving-cert\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.046983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.046972 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-oauth-config\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.047075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.047022 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-config\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.098382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098263 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 20:03:29.098382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098276 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 20:03:29.098382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098273 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 20:03:29.098382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098269 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 20:03:29.098382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098305 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 20:03:29.098382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098344 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 20:03:29.098382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098382 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 20:03:29.098832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.098614 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-877r8\"" Nov 01 20:03:29.148009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.147980 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-config\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148023 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-tmp\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148045 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfxt\" (UniqueName: \"kubernetes.io/projected/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-kube-api-access-xrfxt\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148074 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-service-ca\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148099 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-trusted-ca-bundle\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148136 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-oauth-serving-cert\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148159 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-serving-cert\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148239 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-oauth-config\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148510 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-tmp\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.148755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148728 5633 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:03:29.148913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.148894 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-config\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.149033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.149012 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-oauth-serving-cert\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.149033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.149025 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-service-ca\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.151037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.151021 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-serving-cert\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.151130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.151115 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-oauth-config\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.198410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.198373 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 20:03:29.199155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.199137 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-trusted-ca-bundle\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.199209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.199153 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdbf44876-vjp4b"] Nov 01 20:03:29.205947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.205915 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfxt\" (UniqueName: \"kubernetes.io/projected/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-kube-api-access-xrfxt\") pod \"console-5bdbf44876-vjp4b\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.238371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.238324 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:29.304401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.304360 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:29.426161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.426123 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:29.609083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.608979 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:29.674956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.674873 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdbf44876-vjp4b"] Nov 01 20:03:29.680386 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:29.680355 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e236ab5_4c60_4d6f_b98c_5a9979e0d390.slice/crio-71e4418cdc5a889a85fcc1785006d787a38e169ac275e6742ed9924a904babc0 WatchSource:0}: Error finding container 71e4418cdc5a889a85fcc1785006d787a38e169ac275e6742ed9924a904babc0: Status 404 returned error can't find the container with id 71e4418cdc5a889a85fcc1785006d787a38e169ac275e6742ed9924a904babc0 Nov 01 20:03:29.753122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:29.753084 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:30.598974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:30.598938 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbf44876-vjp4b" event={"ID":"9e236ab5-4c60-4d6f-b98c-5a9979e0d390","Type":"ContainerStarted","Data":"71e4418cdc5a889a85fcc1785006d787a38e169ac275e6742ed9924a904babc0"} Nov 01 20:03:32.735548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:32.735504 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:33.386873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.386833 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:03:33.387658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.387262 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="alertmanager" containerID="cri-o://4a95e061e032a7c307d5a8be838537baacfeac76fd42a9c935c3a055c3a42723" gracePeriod=120 Nov 01 20:03:33.387658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.387297 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-metric" containerID="cri-o://e34ab782299ca56d7d3f3ec1bfb466251def8b2a08aa36e3763d86702b5cdde5" gracePeriod=120 Nov 01 20:03:33.387658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.387296 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy" containerID="cri-o://8b47592d5a1847518a81aa57c20716ebceb0c55e44b57903cb6d3c84a76c82f4" gracePeriod=120 Nov 01 20:03:33.387658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.387370 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="prom-label-proxy" containerID="cri-o://960b1bfed035003faab0ff12c7192c3b87fa0fa54c42cb78f38144a522dd5488" gracePeriod=120 Nov 01 20:03:33.387658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.387467 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-web" containerID="cri-o://9cb53efa96168defab709ed4f51043ee051132eee735bc525927455717a3a938" gracePeriod=120 Nov 01 20:03:33.387658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.387468 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="config-reloader" containerID="cri-o://6f3a2e6476bc9704bcca2c44394a4a97e7e8cdea449f29bc995c7a9070b02e7f" gracePeriod=120 Nov 01 20:03:33.614676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614635 5633 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="960b1bfed035003faab0ff12c7192c3b87fa0fa54c42cb78f38144a522dd5488" exitCode=0 Nov 01 20:03:33.614676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614669 5633 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="8b47592d5a1847518a81aa57c20716ebceb0c55e44b57903cb6d3c84a76c82f4" exitCode=0 Nov 01 20:03:33.614676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614679 5633 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="6f3a2e6476bc9704bcca2c44394a4a97e7e8cdea449f29bc995c7a9070b02e7f" exitCode=0 Nov 01 20:03:33.614676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614687 5633 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="4a95e061e032a7c307d5a8be838537baacfeac76fd42a9c935c3a055c3a42723" exitCode=0 Nov 01 20:03:33.614991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614711 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"960b1bfed035003faab0ff12c7192c3b87fa0fa54c42cb78f38144a522dd5488"} Nov 01 20:03:33.614991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614825 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"8b47592d5a1847518a81aa57c20716ebceb0c55e44b57903cb6d3c84a76c82f4"} Nov 01 20:03:33.614991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614853 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"6f3a2e6476bc9704bcca2c44394a4a97e7e8cdea449f29bc995c7a9070b02e7f"} Nov 01 20:03:33.614991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:33.614881 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"4a95e061e032a7c307d5a8be838537baacfeac76fd42a9c935c3a055c3a42723"} Nov 01 20:03:34.078499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.078462 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:34.252972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.252932 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:34.390846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.390758 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:34.620846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.620819 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbf44876-vjp4b" event={"ID":"9e236ab5-4c60-4d6f-b98c-5a9979e0d390","Type":"ContainerStarted","Data":"731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6"} Nov 01 20:03:34.624186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.624084 5633 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="e34ab782299ca56d7d3f3ec1bfb466251def8b2a08aa36e3763d86702b5cdde5" exitCode=0 Nov 01 20:03:34.624186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.624104 5633 generic.go:356] "Generic (PLEG): container finished" podID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerID="9cb53efa96168defab709ed4f51043ee051132eee735bc525927455717a3a938" exitCode=0 Nov 01 20:03:34.624186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.624159 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"e34ab782299ca56d7d3f3ec1bfb466251def8b2a08aa36e3763d86702b5cdde5"} Nov 01 20:03:34.624293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.624243 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"9cb53efa96168defab709ed4f51043ee051132eee735bc525927455717a3a938"} Nov 01 20:03:34.638677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.638652 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:34.697319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697235 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697275 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697356 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697394 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697524 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697579 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697669 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697709 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697751 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697777 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697807 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697834 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697875 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\") pod \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\" (UID: \"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49\") " Nov 01 20:03:34.697985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.697961 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:03:34.698642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.698359 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:34.698642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.698488 5633 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.698642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.698510 5633 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-alertmanager-main-db\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.700111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.700075 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:34.701115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.701002 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:03:34.701655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.701630 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:34.702176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.702147 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:34.702513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.702492 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:34.702653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.702631 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "kube-api-access-qfggb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:03:34.702957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.702933 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:03:34.703154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.703137 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:34.703848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.703826 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:34.706647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.706608 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:34.713139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.713113 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config" pod "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" (UID: "8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:34.736526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.736463 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5bdbf44876-vjp4b" podStartSLOduration=2.743663715 podStartE2EDuration="6.736439218s" totalImagesPullingTime="3.992775503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:34.735917288 +0000 UTC m=+22.410176543" watchObservedRunningTime="2026-11-01 20:03:34.736439218 +0000 UTC m=+22.410698523" Nov 01 20:03:34.799384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799354 5633 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-out\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799379 5633 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799389 5633 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-cluster-tls-config\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799398 5633 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799408 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfggb\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-kube-api-access-qfggb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799417 5633 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-metrics-client-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799428 5633 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799437 5633 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799446 5633 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-web-config\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799455 5633 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-tls-assets\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:34.799618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:34.799464 5633 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49-secret-alertmanager-main-tls\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:35.631440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.631397 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49","Type":"ContainerDied","Data":"6d86f56c568c4d7cd2414606fe536668a6068b0b61396c31c997be701c726ab4"} Nov 01 20:03:35.631895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.631479 5633 scope.go:122] "RemoveContainer" containerID="960b1bfed035003faab0ff12c7192c3b87fa0fa54c42cb78f38144a522dd5488" Nov 01 20:03:35.631895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.631609 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:35.640058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.640034 5633 scope.go:122] "RemoveContainer" containerID="e34ab782299ca56d7d3f3ec1bfb466251def8b2a08aa36e3763d86702b5cdde5" Nov 01 20:03:35.647438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.647416 5633 scope.go:122] "RemoveContainer" containerID="8b47592d5a1847518a81aa57c20716ebceb0c55e44b57903cb6d3c84a76c82f4" Nov 01 20:03:35.655053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.655029 5633 scope.go:122] "RemoveContainer" containerID="9cb53efa96168defab709ed4f51043ee051132eee735bc525927455717a3a938" Nov 01 20:03:35.662384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.662364 5633 scope.go:122] "RemoveContainer" containerID="6f3a2e6476bc9704bcca2c44394a4a97e7e8cdea449f29bc995c7a9070b02e7f" Nov 01 20:03:35.669672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.669652 5633 scope.go:122] "RemoveContainer" containerID="4a95e061e032a7c307d5a8be838537baacfeac76fd42a9c935c3a055c3a42723" Nov 01 20:03:35.677478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.677458 5633 scope.go:122] "RemoveContainer" containerID="712284bdce2a4f3561ca2a1b4288fc6a91da6ef981e35e8968dc028b7151ed58" Nov 01 20:03:35.845499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.845415 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:03:35.895492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.895462 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:03:35.895966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.895941 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:03:35.896249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896232 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="prom-label-proxy" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896250 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="prom-label-proxy" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896263 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="config-reloader" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896268 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="config-reloader" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896280 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-metric" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896285 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-metric" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896293 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-web" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896299 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-web" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896307 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="alertmanager" Nov 01 20:03:35.896312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896314 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="alertmanager" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896348 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896356 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896367 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="init-config-reloader" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896372 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="init-config-reloader" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896436 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-metric" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896448 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy-web" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896455 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="alertmanager" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896472 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="kube-rbac-proxy" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896479 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="init-config-reloader" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896488 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="prom-label-proxy" Nov 01 20:03:35.896798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.896495 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" containerName="config-reloader" Nov 01 20:03:35.900065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.900044 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:35.955404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.955202 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 20:03:35.955404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.955308 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 20:03:35.956219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.955722 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 20:03:35.956219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.955807 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 20:03:35.956219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.955906 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 20:03:35.956219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:35.956005 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 20:03:36.001534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.001503 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 20:03:36.001698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.001606 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 20:03:36.001698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.001652 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ljq7\"" Nov 01 20:03:36.001698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.001652 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 20:03:36.009665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.009637 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbszv\" (UniqueName: \"kubernetes.io/projected/519cd664-61e6-4b51-bd44-8a3bac94dbe1-kube-api-access-lbszv\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.009766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.009684 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.009766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.009757 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/519cd664-61e6-4b51-bd44-8a3bac94dbe1-config-out\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.009828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.009787 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-config-volume\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.009828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.009808 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.009923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.009900 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.009960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.009946 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/519cd664-61e6-4b51-bd44-8a3bac94dbe1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.010041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.010025 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/519cd664-61e6-4b51-bd44-8a3bac94dbe1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.010082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.010053 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.010082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.010073 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-web-config\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.010169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.010154 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/519cd664-61e6-4b51-bd44-8a3bac94dbe1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.010206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.010178 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/519cd664-61e6-4b51-bd44-8a3bac94dbe1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.010206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.010198 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111605 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/519cd664-61e6-4b51-bd44-8a3bac94dbe1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111654 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111679 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-web-config\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111722 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/519cd664-61e6-4b51-bd44-8a3bac94dbe1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111761 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/519cd664-61e6-4b51-bd44-8a3bac94dbe1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111788 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111819 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbszv\" (UniqueName: \"kubernetes.io/projected/519cd664-61e6-4b51-bd44-8a3bac94dbe1-kube-api-access-lbszv\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111926 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.111978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.111975 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/519cd664-61e6-4b51-bd44-8a3bac94dbe1-config-out\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.112258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.112002 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-config-volume\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.112258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.112037 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.112258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.112068 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.112258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.112100 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/519cd664-61e6-4b51-bd44-8a3bac94dbe1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.113134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.112679 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/519cd664-61e6-4b51-bd44-8a3bac94dbe1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.113134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.113057 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/519cd664-61e6-4b51-bd44-8a3bac94dbe1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.113134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.113086 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/519cd664-61e6-4b51-bd44-8a3bac94dbe1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.115410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.115384 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-web-config\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.115613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.115585 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.115842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.115815 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.115914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.115863 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-config-volume\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.115914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.115886 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.116068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.115891 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/519cd664-61e6-4b51-bd44-8a3bac94dbe1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.116068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.116045 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.117006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.116983 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/519cd664-61e6-4b51-bd44-8a3bac94dbe1-config-out\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.117530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.117513 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/519cd664-61e6-4b51-bd44-8a3bac94dbe1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.129030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.128997 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:03:36.171801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.171752 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbszv\" (UniqueName: \"kubernetes.io/projected/519cd664-61e6-4b51-bd44-8a3bac94dbe1-kube-api-access-lbszv\") pod \"alertmanager-main-0\" (UID: \"519cd664-61e6-4b51-bd44-8a3bac94dbe1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.215776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.215738 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 20:03:36.464512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.464476 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49" path="/var/lib/kubelet/pods/8a1f68dc-829d-48d3-bfe8-71e7ca8f5b49/volumes" Nov 01 20:03:36.643131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:36.643100 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 20:03:36.647250 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:36.647221 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519cd664_61e6_4b51_bd44_8a3bac94dbe1.slice/crio-3c4a706132ac2e9b7eb716b17934a4376ba5621720b5ad9defd5733996023ceb WatchSource:0}: Error finding container 3c4a706132ac2e9b7eb716b17934a4376ba5621720b5ad9defd5733996023ceb: Status 404 returned error can't find the container with id 3c4a706132ac2e9b7eb716b17934a4376ba5621720b5ad9defd5733996023ceb Nov 01 20:03:37.641273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:37.641240 5633 generic.go:356] "Generic (PLEG): container finished" podID="519cd664-61e6-4b51-bd44-8a3bac94dbe1" containerID="dbacc66dbe6dfa196432c859f0fa03e1072570a9e95c27389da0a15023682b66" exitCode=0 Nov 01 20:03:37.641448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:37.641296 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerDied","Data":"dbacc66dbe6dfa196432c859f0fa03e1072570a9e95c27389da0a15023682b66"} Nov 01 20:03:37.641448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:37.641374 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerStarted","Data":"3c4a706132ac2e9b7eb716b17934a4376ba5621720b5ad9defd5733996023ceb"} Nov 01 20:03:38.653779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:38.653743 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerStarted","Data":"653bbd10f34c49a95df7cbb6c0ca8f8cfa42aecd2d3fcdbe5e7f2646f8505f86"} Nov 01 20:03:38.654150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:38.653833 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerStarted","Data":"68f9ce4e9c1ae5c445c86999336c291e8f65f2891bc963cef9647a560857a9b8"} Nov 01 20:03:38.654150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:38.653864 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerStarted","Data":"f7c4d19b147f9f4f13d5904a0f6343c84188630f6604aa9df7254fa3a8f3197a"} Nov 01 20:03:38.654150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:38.653889 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerStarted","Data":"75b1b4922d42ba75d499abef165ee4b09cacad56b7b6f8b6e700c0fc17479cc7"} Nov 01 20:03:38.654150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:38.653911 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerStarted","Data":"6525fdc8f817bf7ad7ceae7091a21e8de81a41e60e5cf3799083d869aadf7b68"} Nov 01 20:03:38.654150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:38.653933 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"519cd664-61e6-4b51-bd44-8a3bac94dbe1","Type":"ContainerStarted","Data":"027273299a062ebf6052bf4a531cd094e6b445cfe02e0cc1f5bbbfb7774a16e0"} Nov 01 20:03:38.776973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:38.776915 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.76502692 podStartE2EDuration="3.776900351s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.873431ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:38.776096903 +0000 UTC m=+26.450356160" watchObservedRunningTime="2026-11-01 20:03:38.776900351 +0000 UTC m=+26.451159673" Nov 01 20:03:39.118087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.118053 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:39.118423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.118387 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="prometheus" containerID="cri-o://a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" gracePeriod=600 Nov 01 20:03:39.118686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.118387 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy" containerID="cri-o://eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" gracePeriod=600 Nov 01 20:03:39.118785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.118444 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="config-reloader" containerID="cri-o://a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" gracePeriod=600 Nov 01 20:03:39.118857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.118445 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="thanos-sidecar" containerID="cri-o://e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" gracePeriod=600 Nov 01 20:03:39.118942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.118436 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-thanos" containerID="cri-o://af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" gracePeriod=600 Nov 01 20:03:39.118942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.118459 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-web" containerID="cri-o://139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" gracePeriod=600 Nov 01 20:03:39.156065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.156018 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:39.304652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.304608 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:39.304837 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.304687 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:39.310777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.310753 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:39.390540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.390440 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:39.402758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.402734 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.440872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.440837 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.440872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.440878 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.440899 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.440927 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.440958 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.440998 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441035 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441065 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441091 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441133 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441167 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441199 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441226 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441262 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441355 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441371 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441389 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441421 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441441 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.441517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441505 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.442223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441551 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.442223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.441583 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\") pod \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\" (UID: \"0fab5652-b1da-42d8-9c42-1e32ad4fc054\") " Nov 01 20:03:39.442355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.442273 5633 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.442355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.442296 5633 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.442707 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.442678 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:39.443941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.442860 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:39.443941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.442899 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:39.443941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.443597 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:39.444722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.444320 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:03:39.444722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.444555 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:39.445293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.445252 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.445428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.445349 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "kube-api-access-gjxds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:03:39.445726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.445686 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.446059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.446029 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.446141 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.446108 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:03:39.446201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.446167 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.446393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.446252 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.446393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.446305 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.446541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.446453 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.446939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.446920 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:03:39.447817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.447792 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.457320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.457291 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config" pod "0fab5652-b1da-42d8-9c42-1e32ad4fc054" (UID: "0fab5652-b1da-42d8-9c42-1e32ad4fc054"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:39.543056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543020 5633 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543049 5633 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543061 5633 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543070 5633 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-configmap-metrics-client-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543079 5633 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543088 5633 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-db\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543097 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjxds\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-kube-api-access-gjxds\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543106 5633 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543114 5633 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-metrics-client-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543123 5633 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543132 5633 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fab5652-b1da-42d8-9c42-1e32ad4fc054-config-out\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543140 5633 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-kube-rbac-proxy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543150 5633 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543159 5633 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543167 5633 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fab5652-b1da-42d8-9c42-1e32ad4fc054-tls-assets\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543175 5633 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-secret-grpc-tls\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543184 5633 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fab5652-b1da-42d8-9c42-1e32ad4fc054-web-config\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.543299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.543194 5633 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fab5652-b1da-42d8-9c42-1e32ad4fc054-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:39.664047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.663965 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" exitCode=0 Nov 01 20:03:39.664047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.663990 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" exitCode=0 Nov 01 20:03:39.664047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.663997 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" exitCode=0 Nov 01 20:03:39.664047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664003 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" exitCode=0 Nov 01 20:03:39.664047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664008 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" exitCode=0 Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664016 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" exitCode=0 Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664056 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664046 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664292 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664325 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664365 5633 scope.go:122] "RemoveContainer" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664381 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664421 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352"} Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664459 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107"} Nov 01 20:03:39.664642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.664507 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fab5652-b1da-42d8-9c42-1e32ad4fc054","Type":"ContainerDied","Data":"7769919e3bf356610a7f260d3e123702ea743fabbd479d2bf9b062bd67661add"} Nov 01 20:03:39.668928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.668903 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:03:39.674294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.674275 5633 scope.go:122] "RemoveContainer" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.681804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.681784 5633 scope.go:122] "RemoveContainer" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.688981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.688958 5633 scope.go:122] "RemoveContainer" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.696297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.696277 5633 scope.go:122] "RemoveContainer" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.703722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.703702 5633 scope.go:122] "RemoveContainer" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.711642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.711618 5633 scope.go:122] "RemoveContainer" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.719715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.719696 5633 scope.go:122] "RemoveContainer" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.720025 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:39.720003 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": container with ID starting with af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef not found: ID does not exist" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.720099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720034 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} err="failed to get container status \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": rpc error: code = NotFound desc = could not find container \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": container with ID starting with af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef not found: ID does not exist" Nov 01 20:03:39.720099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720070 5633 scope.go:122] "RemoveContainer" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.720367 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:39.720341 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": container with ID starting with eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6 not found: ID does not exist" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.720456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720373 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} err="failed to get container status \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": rpc error: code = NotFound desc = could not find container \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": container with ID starting with eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6 not found: ID does not exist" Nov 01 20:03:39.720456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720393 5633 scope.go:122] "RemoveContainer" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.720657 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:39.720638 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": container with ID starting with 139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0 not found: ID does not exist" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.720721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720665 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} err="failed to get container status \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": rpc error: code = NotFound desc = could not find container \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": container with ID starting with 139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0 not found: ID does not exist" Nov 01 20:03:39.720721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720685 5633 scope.go:122] "RemoveContainer" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.720901 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:39.720886 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": container with ID starting with e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d not found: ID does not exist" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.720960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720907 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} err="failed to get container status \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": rpc error: code = NotFound desc = could not find container \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": container with ID starting with e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d not found: ID does not exist" Nov 01 20:03:39.720960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.720927 5633 scope.go:122] "RemoveContainer" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.721162 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:39.721146 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": container with ID starting with a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352 not found: ID does not exist" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.721203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721166 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352"} err="failed to get container status \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": rpc error: code = NotFound desc = could not find container \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": container with ID starting with a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352 not found: ID does not exist" Nov 01 20:03:39.721203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721181 5633 scope.go:122] "RemoveContainer" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.721450 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:39.721427 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": container with ID starting with a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107 not found: ID does not exist" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.721541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721453 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107"} err="failed to get container status \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": rpc error: code = NotFound desc = could not find container \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": container with ID starting with a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107 not found: ID does not exist" Nov 01 20:03:39.721541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721468 5633 scope.go:122] "RemoveContainer" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.721730 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:39.721713 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": container with ID starting with cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9 not found: ID does not exist" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.721775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721735 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9"} err="failed to get container status \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": rpc error: code = NotFound desc = could not find container \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": container with ID starting with cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9 not found: ID does not exist" Nov 01 20:03:39.721775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721746 5633 scope.go:122] "RemoveContainer" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.721977 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721960 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} err="failed to get container status \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": rpc error: code = NotFound desc = could not find container \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": container with ID starting with af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef not found: ID does not exist" Nov 01 20:03:39.722019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.721978 5633 scope.go:122] "RemoveContainer" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.722192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722176 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} err="failed to get container status \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": rpc error: code = NotFound desc = could not find container \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": container with ID starting with eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6 not found: ID does not exist" Nov 01 20:03:39.722234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722193 5633 scope.go:122] "RemoveContainer" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.722432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722415 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} err="failed to get container status \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": rpc error: code = NotFound desc = could not find container \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": container with ID starting with 139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0 not found: ID does not exist" Nov 01 20:03:39.722485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722432 5633 scope.go:122] "RemoveContainer" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.722657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722638 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} err="failed to get container status \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": rpc error: code = NotFound desc = could not find container \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": container with ID starting with e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d not found: ID does not exist" Nov 01 20:03:39.722712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722656 5633 scope.go:122] "RemoveContainer" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.722878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722851 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352"} err="failed to get container status \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": rpc error: code = NotFound desc = could not find container \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": container with ID starting with a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352 not found: ID does not exist" Nov 01 20:03:39.722923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.722877 5633 scope.go:122] "RemoveContainer" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.723084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723067 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107"} err="failed to get container status \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": rpc error: code = NotFound desc = could not find container \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": container with ID starting with a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107 not found: ID does not exist" Nov 01 20:03:39.723084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723083 5633 scope.go:122] "RemoveContainer" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.723293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723272 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9"} err="failed to get container status \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": rpc error: code = NotFound desc = could not find container \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": container with ID starting with cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9 not found: ID does not exist" Nov 01 20:03:39.723370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723294 5633 scope.go:122] "RemoveContainer" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.723517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723502 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} err="failed to get container status \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": rpc error: code = NotFound desc = could not find container \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": container with ID starting with af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef not found: ID does not exist" Nov 01 20:03:39.723580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723517 5633 scope.go:122] "RemoveContainer" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.723734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723717 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} err="failed to get container status \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": rpc error: code = NotFound desc = could not find container \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": container with ID starting with eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6 not found: ID does not exist" Nov 01 20:03:39.723829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723733 5633 scope.go:122] "RemoveContainer" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.723917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723900 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} err="failed to get container status \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": rpc error: code = NotFound desc = could not find container \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": container with ID starting with 139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0 not found: ID does not exist" Nov 01 20:03:39.723956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.723919 5633 scope.go:122] "RemoveContainer" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.724126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724109 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} err="failed to get container status \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": rpc error: code = NotFound desc = could not find container \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": container with ID starting with e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d not found: ID does not exist" Nov 01 20:03:39.724126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724124 5633 scope.go:122] "RemoveContainer" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.724296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724277 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352"} err="failed to get container status \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": rpc error: code = NotFound desc = could not find container \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": container with ID starting with a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352 not found: ID does not exist" Nov 01 20:03:39.724362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724297 5633 scope.go:122] "RemoveContainer" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.724520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724504 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107"} err="failed to get container status \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": rpc error: code = NotFound desc = could not find container \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": container with ID starting with a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107 not found: ID does not exist" Nov 01 20:03:39.724575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724521 5633 scope.go:122] "RemoveContainer" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.724758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724739 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9"} err="failed to get container status \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": rpc error: code = NotFound desc = could not find container \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": container with ID starting with cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9 not found: ID does not exist" Nov 01 20:03:39.724808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724760 5633 scope.go:122] "RemoveContainer" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.724968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724953 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} err="failed to get container status \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": rpc error: code = NotFound desc = could not find container \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": container with ID starting with af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef not found: ID does not exist" Nov 01 20:03:39.725017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.724968 5633 scope.go:122] "RemoveContainer" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.725164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725150 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} err="failed to get container status \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": rpc error: code = NotFound desc = could not find container \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": container with ID starting with eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6 not found: ID does not exist" Nov 01 20:03:39.725202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725164 5633 scope.go:122] "RemoveContainer" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.725371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725351 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} err="failed to get container status \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": rpc error: code = NotFound desc = could not find container \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": container with ID starting with 139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0 not found: ID does not exist" Nov 01 20:03:39.725411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725372 5633 scope.go:122] "RemoveContainer" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.725588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725563 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} err="failed to get container status \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": rpc error: code = NotFound desc = could not find container \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": container with ID starting with e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d not found: ID does not exist" Nov 01 20:03:39.725630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725588 5633 scope.go:122] "RemoveContainer" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.725774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725756 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352"} err="failed to get container status \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": rpc error: code = NotFound desc = could not find container \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": container with ID starting with a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352 not found: ID does not exist" Nov 01 20:03:39.725816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725774 5633 scope.go:122] "RemoveContainer" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.725961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725946 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107"} err="failed to get container status \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": rpc error: code = NotFound desc = could not find container \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": container with ID starting with a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107 not found: ID does not exist" Nov 01 20:03:39.726028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.725961 5633 scope.go:122] "RemoveContainer" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.726160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726141 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9"} err="failed to get container status \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": rpc error: code = NotFound desc = could not find container \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": container with ID starting with cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9 not found: ID does not exist" Nov 01 20:03:39.726199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726161 5633 scope.go:122] "RemoveContainer" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.726415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726398 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} err="failed to get container status \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": rpc error: code = NotFound desc = could not find container \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": container with ID starting with af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef not found: ID does not exist" Nov 01 20:03:39.726479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726415 5633 scope.go:122] "RemoveContainer" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.726651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726635 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} err="failed to get container status \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": rpc error: code = NotFound desc = could not find container \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": container with ID starting with eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6 not found: ID does not exist" Nov 01 20:03:39.726715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726651 5633 scope.go:122] "RemoveContainer" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.726889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726872 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} err="failed to get container status \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": rpc error: code = NotFound desc = could not find container \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": container with ID starting with 139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0 not found: ID does not exist" Nov 01 20:03:39.726939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.726889 5633 scope.go:122] "RemoveContainer" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.727143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727126 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} err="failed to get container status \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": rpc error: code = NotFound desc = could not find container \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": container with ID starting with e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d not found: ID does not exist" Nov 01 20:03:39.727182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727144 5633 scope.go:122] "RemoveContainer" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.727415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727396 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352"} err="failed to get container status \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": rpc error: code = NotFound desc = could not find container \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": container with ID starting with a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352 not found: ID does not exist" Nov 01 20:03:39.727415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727415 5633 scope.go:122] "RemoveContainer" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.727644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727627 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107"} err="failed to get container status \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": rpc error: code = NotFound desc = could not find container \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": container with ID starting with a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107 not found: ID does not exist" Nov 01 20:03:39.727691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727645 5633 scope.go:122] "RemoveContainer" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.727854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727835 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9"} err="failed to get container status \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": rpc error: code = NotFound desc = could not find container \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": container with ID starting with cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9 not found: ID does not exist" Nov 01 20:03:39.727919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.727853 5633 scope.go:122] "RemoveContainer" containerID="af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef" Nov 01 20:03:39.728082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728065 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef"} err="failed to get container status \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": rpc error: code = NotFound desc = could not find container \"af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef\": container with ID starting with af79d01f4fd37447bbe3ddb06fbf6d3ee948fcbec1e589eacb637f2545f8f6ef not found: ID does not exist" Nov 01 20:03:39.728136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728083 5633 scope.go:122] "RemoveContainer" containerID="eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6" Nov 01 20:03:39.728289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728274 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6"} err="failed to get container status \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": rpc error: code = NotFound desc = could not find container \"eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6\": container with ID starting with eadbb9b9dfb8e1b5ec9c8ce2e477987c62b0a3b25100213a767adc2dbcb9bbe6 not found: ID does not exist" Nov 01 20:03:39.728289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728288 5633 scope.go:122] "RemoveContainer" containerID="139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0" Nov 01 20:03:39.728538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728520 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0"} err="failed to get container status \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": rpc error: code = NotFound desc = could not find container \"139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0\": container with ID starting with 139535585efd42447104d18ac8d2b22bfecad156545f58626ba72e7e77b237f0 not found: ID does not exist" Nov 01 20:03:39.728580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728539 5633 scope.go:122] "RemoveContainer" containerID="e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d" Nov 01 20:03:39.728727 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728711 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d"} err="failed to get container status \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": rpc error: code = NotFound desc = could not find container \"e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d\": container with ID starting with e5bd6053aba4fe78374e8b62296e07ca1197825443d20b5c0d565bd1ba85a92d not found: ID does not exist" Nov 01 20:03:39.728768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728727 5633 scope.go:122] "RemoveContainer" containerID="a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352" Nov 01 20:03:39.728943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728927 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352"} err="failed to get container status \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": rpc error: code = NotFound desc = could not find container \"a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352\": container with ID starting with a572b1a7d08f550bbcccbe43923fce3a7dec45cd6a1c6101394b7fdbb4955352 not found: ID does not exist" Nov 01 20:03:39.728943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.728942 5633 scope.go:122] "RemoveContainer" containerID="a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107" Nov 01 20:03:39.729147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.729130 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107"} err="failed to get container status \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": rpc error: code = NotFound desc = could not find container \"a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107\": container with ID starting with a39f41caf808e31101b0093f230ebbd0ed6b53d9a11372ee42d11b981d621107 not found: ID does not exist" Nov 01 20:03:39.729147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.729146 5633 scope.go:122] "RemoveContainer" containerID="cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9" Nov 01 20:03:39.729366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.729349 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9"} err="failed to get container status \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": rpc error: code = NotFound desc = could not find container \"cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9\": container with ID starting with cdf03397126b228b711c1fdc506a571cd9361df8e1d3fd4442522ed2403422b9 not found: ID does not exist" Nov 01 20:03:39.832014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.831980 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:39.834974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.834946 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:39.860508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860484 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:39.860760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860748 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy" Nov 01 20:03:39.860797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860761 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy" Nov 01 20:03:39.860797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860771 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-thanos" Nov 01 20:03:39.860797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860777 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-thanos" Nov 01 20:03:39.860797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860788 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="thanos-sidecar" Nov 01 20:03:39.860797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860793 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="thanos-sidecar" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860802 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="init-config-reloader" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860807 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="init-config-reloader" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860816 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-web" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860821 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-web" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860832 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="config-reloader" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860837 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="config-reloader" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860843 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="prometheus" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860847 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="prometheus" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860888 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860896 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-thanos" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860904 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="thanos-sidecar" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860910 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="kube-rbac-proxy-web" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860915 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="config-reloader" Nov 01 20:03:39.860944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.860920 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" containerName="prometheus" Nov 01 20:03:39.866075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.866054 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.918919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.918824 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 20:03:39.918919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.918826 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 20:03:39.918919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.918860 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 20:03:39.918919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.918909 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.918922 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ft87h\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.918909 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.918944 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.919053 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.919099 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.919133 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.919147 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c9o03dg9n84k5\"" Nov 01 20:03:39.919227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.919171 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 20:03:39.919496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.919437 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 20:03:39.919496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.919474 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 20:03:39.947646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.947609 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.947646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.947647 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.947863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.947673 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.947863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.947727 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.947980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.947868 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.947980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.947905 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948006 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948033 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948059 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948120 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948144 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948210 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948262 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948297 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948354 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948385 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948415 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948457 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948487 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqg7\" (UniqueName: \"kubernetes.io/projected/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-kube-api-access-7xqg7\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.948567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.948504 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-config\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:39.969415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.969380 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 20:03:39.969585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.969433 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 20:03:39.971415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:39.971388 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:40.049879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.049839 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqg7\" (UniqueName: \"kubernetes.io/projected/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-kube-api-access-7xqg7\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.049879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.049879 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-config\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.049901 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050041 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050106 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050134 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050172 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050209 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050251 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050276 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050301 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050354 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050387 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050413 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050443 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050473 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050507 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050533 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050566 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050585 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.050732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050643 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.051134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050730 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.051134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.050835 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.051430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.051168 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.051602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.051578 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.053419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.052435 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.053419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.053076 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.053419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.053749 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-config\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.053419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.054146 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.053419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.054512 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.053419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.054664 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.057178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.056320 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.057178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.056764 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.057178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.056739 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.057178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.056976 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.057422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.057176 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.057461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.057445 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-web-config\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.059414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.059391 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.059583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.059565 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-config-out\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.107006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.106964 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqg7\" (UniqueName: \"kubernetes.io/projected/4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6-kube-api-access-7xqg7\") pod \"prometheus-k8s-0\" (UID: \"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.192099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.192017 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:40.464373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.464325 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fab5652-b1da-42d8-9c42-1e32ad4fc054" path="/var/lib/kubelet/pods/0fab5652-b1da-42d8-9c42-1e32ad4fc054/volumes" Nov 01 20:03:40.618860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.618829 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 20:03:40.621718 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:03:40.621692 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4407a2d2_08d0_4c4a_b8bd_509a76bf2fc6.slice/crio-c6b4f3942d34230358d88f4706f3f2dc626194e5c0e2989c64b560e4e02e204e WatchSource:0}: Error finding container c6b4f3942d34230358d88f4706f3f2dc626194e5c0e2989c64b560e4e02e204e: Status 404 returned error can't find the container with id c6b4f3942d34230358d88f4706f3f2dc626194e5c0e2989c64b560e4e02e204e Nov 01 20:03:40.669703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:40.669669 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerStarted","Data":"c6b4f3942d34230358d88f4706f3f2dc626194e5c0e2989c64b560e4e02e204e"} Nov 01 20:03:41.676187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:41.676157 5633 generic.go:356] "Generic (PLEG): container finished" podID="4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6" containerID="5b29fed0519e2e0635a79a9c1608025f72c8660ce93e92e043daa67a4999fcfa" exitCode=0 Nov 01 20:03:41.676187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:41.676193 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerDied","Data":"5b29fed0519e2e0635a79a9c1608025f72c8660ce93e92e043daa67a4999fcfa"} Nov 01 20:03:42.685951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.685916 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerStarted","Data":"58c3f61518c072ce6df364fb3fd30ca689cf1e52685b83fc202dc274330ef0ba"} Nov 01 20:03:42.686463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.685993 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerStarted","Data":"8968f933158f2afc9aa355f65cad5add70aaeb36ee35fb677bdd491a2b8a1658"} Nov 01 20:03:42.686463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.686052 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerStarted","Data":"3714b28179005c6b2b0fbdf7a25b44bdd3d9e019340972879ba524ab18786d3d"} Nov 01 20:03:42.686463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.686093 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerStarted","Data":"215b5e2b111bc0136582797bbd1338addc9ff41bd5d4fc20e667fbcd099a3eb1"} Nov 01 20:03:42.686463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.686140 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerStarted","Data":"4238d821335befb9a160fd00270e5abba5e08a890209ecabc6239325f2722bb1"} Nov 01 20:03:42.686463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.686177 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6","Type":"ContainerStarted","Data":"3cfd6c78311706866e0959c6ef8ba2a93eb0669ed7d57cbc5b6321c7280dc211"} Nov 01 20:03:42.808610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.808552 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.7931776680000002 podStartE2EDuration="3.808537295s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.359627ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:42.807765241 +0000 UTC m=+30.482024497" watchObservedRunningTime="2026-11-01 20:03:42.808537295 +0000 UTC m=+30.482796544" Nov 01 20:03:42.963346 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:42.963308 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:43.087314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:43.087280 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:03:43.627568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:43.627539 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ks9th" Nov 01 20:03:45.192495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:45.192450 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:03:46.155253 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:03:46.155216 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:03:47.083010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:47.082979 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:03:47.086832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:03:47.086810 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d569ccc7c-67pl4" Nov 01 20:04:16.155108 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:04:16.155069 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:04:17.526021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:04:17.525985 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:04:38.051733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:04:38.051693 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:04:38.068449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:04:38.068411 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:04:38.899670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:04:38.899641 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:04:46.155969 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:04:46.155931 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:05:16.155469 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:05:16.155427 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:05:17.850893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:17.850850 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:05:46.155168 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:05:46.155124 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:05:54.330870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:54.330829 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:05:54.498373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:54.498312 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:05:54.633354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:54.633246 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:05:55.736010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:55.735963 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:05:55.891881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:55.891843 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:05:55.957200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:55.957167 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd"] Nov 01 20:05:55.960596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:55.960566 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.033997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.033918 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvjk\" (UniqueName: \"kubernetes.io/projected/255344a5-c362-4eda-8e42-84e6bdd791bb-kube-api-access-bpvjk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.033997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.033971 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.034187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.034058 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.062015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.061976 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 20:05:56.062015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.061976 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 20:05:56.062236 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.062109 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-v42cj\"" Nov 01 20:05:56.062891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.062871 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd"] Nov 01 20:05:56.134751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.134713 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvjk\" (UniqueName: \"kubernetes.io/projected/255344a5-c362-4eda-8e42-84e6bdd791bb-kube-api-access-bpvjk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.134931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.134777 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.134931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.134802 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.135127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.135108 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.135164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.135147 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.191997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.191961 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvjk\" (UniqueName: \"kubernetes.io/projected/255344a5-c362-4eda-8e42-84e6bdd791bb-kube-api-access-bpvjk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.276001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.275960 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:05:56.643817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.643739 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd"] Nov 01 20:05:56.678294 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:05:56.678264 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255344a5_c362_4eda_8e42_84e6bdd791bb.slice/crio-429d7a60c628fa7d490736e12ff67efa830594f494c9db2013b8abe0127e3cac WatchSource:0}: Error finding container 429d7a60c628fa7d490736e12ff67efa830594f494c9db2013b8abe0127e3cac: Status 404 returned error can't find the container with id 429d7a60c628fa7d490736e12ff67efa830594f494c9db2013b8abe0127e3cac Nov 01 20:05:56.817798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:56.817758 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:05:57.140059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.140018 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7"] Nov 01 20:05:57.144774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.144745 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.156366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.156318 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" event={"ID":"255344a5-c362-4eda-8e42-84e6bdd791bb","Type":"ContainerStarted","Data":"429d7a60c628fa7d490736e12ff67efa830594f494c9db2013b8abe0127e3cac"} Nov 01 20:05:57.199483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.199445 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fkqpn\"" Nov 01 20:05:57.200544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.200515 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7"] Nov 01 20:05:57.246101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.246065 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9btn\" (UniqueName: \"kubernetes.io/projected/ee343e4a-952a-44d4-8ca8-7f99975f1208-kube-api-access-r9btn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.246273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.246110 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.246273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.246131 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.347445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.347412 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9btn\" (UniqueName: \"kubernetes.io/projected/ee343e4a-952a-44d4-8ca8-7f99975f1208-kube-api-access-r9btn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.347640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.347477 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.347716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.347638 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.347938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.347914 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.348007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.347992 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.404611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.404536 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9btn\" (UniqueName: \"kubernetes.io/projected/ee343e4a-952a-44d4-8ca8-7f99975f1208-kube-api-access-r9btn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.462765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.462726 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:05:57.831995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:57.831908 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7"] Nov 01 20:05:57.836215 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:05:57.836185 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee343e4a_952a_44d4_8ca8_7f99975f1208.slice/crio-378a0430eb6dda6ba25b18025894b4e0b4ef2ddf4e26026c127d0d8c738f6fba WatchSource:0}: Error finding container 378a0430eb6dda6ba25b18025894b4e0b4ef2ddf4e26026c127d0d8c738f6fba: Status 404 returned error can't find the container with id 378a0430eb6dda6ba25b18025894b4e0b4ef2ddf4e26026c127d0d8c738f6fba Nov 01 20:05:58.162525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:05:58.162437 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" event={"ID":"ee343e4a-952a-44d4-8ca8-7f99975f1208","Type":"ContainerStarted","Data":"378a0430eb6dda6ba25b18025894b4e0b4ef2ddf4e26026c127d0d8c738f6fba"} Nov 01 20:06:04.188478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.188431 5633 generic.go:356] "Generic (PLEG): container finished" podID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerID="9c8d5d3d39b71c55f7b8ca98ac1b783e8600a4d6b34fb7f916e619ad8dca9df7" exitCode=0 Nov 01 20:06:04.188967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.188484 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" event={"ID":"255344a5-c362-4eda-8e42-84e6bdd791bb","Type":"ContainerDied","Data":"9c8d5d3d39b71c55f7b8ca98ac1b783e8600a4d6b34fb7f916e619ad8dca9df7"} Nov 01 20:06:04.840874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.840840 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc"] Nov 01 20:06:04.844145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.844127 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:04.868775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.868745 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss"] Nov 01 20:06:04.872126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.872108 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:04.896867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.896838 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds"] Nov 01 20:06:04.900126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.900107 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:04.917787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.917758 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2kfch\"" Nov 01 20:06:04.918521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.918498 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc"] Nov 01 20:06:04.924133 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.924110 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:04.924278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.924143 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczct\" (UniqueName: \"kubernetes.io/projected/4b2dfcef-ceb9-4518-96a3-c68b04071626-kube-api-access-dczct\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:04.924278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.924172 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:04.946246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.946215 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j"] Nov 01 20:06:04.949667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.949646 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:04.956958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.956932 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds"] Nov 01 20:06:04.993367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.993318 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j"] Nov 01 20:06:04.999274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:04.999244 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss"] Nov 01 20:06:05.003028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.002991 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-qhjhf\"" Nov 01 20:06:05.003186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.003068 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-vndjh\"" Nov 01 20:06:05.025473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025439 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczct\" (UniqueName: \"kubernetes.io/projected/4b2dfcef-ceb9-4518-96a3-c68b04071626-kube-api-access-dczct\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:05.025640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025482 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:05.025640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025503 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.025640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025543 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.025640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025583 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.025640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025610 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.025827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025719 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.025827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025762 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4g4\" (UniqueName: \"kubernetes.io/projected/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-kube-api-access-zk4g4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.025894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025829 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.025894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025836 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:05.025894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025883 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5nd\" (UniqueName: \"kubernetes.io/projected/39bcb846-ac58-4cc0-905a-be3408373098-kube-api-access-2d5nd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.025983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025926 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:05.025983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.025951 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j4x8\" (UniqueName: \"kubernetes.io/projected/ca05f496-f0d3-4c51-a11c-6874c7b37171-kube-api-access-9j4x8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.026229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.026212 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:05.082204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.082168 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczct\" (UniqueName: \"kubernetes.io/projected/4b2dfcef-ceb9-4518-96a3-c68b04071626-kube-api-access-dczct\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:05.127382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127270 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.127382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127313 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.127382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127354 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.127382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127378 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.127720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127413 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4g4\" (UniqueName: \"kubernetes.io/projected/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-kube-api-access-zk4g4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.127720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127445 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.127720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127475 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5nd\" (UniqueName: \"kubernetes.io/projected/39bcb846-ac58-4cc0-905a-be3408373098-kube-api-access-2d5nd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.127720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127524 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j4x8\" (UniqueName: \"kubernetes.io/projected/ca05f496-f0d3-4c51-a11c-6874c7b37171-kube-api-access-9j4x8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.127720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127566 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.127968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127799 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.127968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127859 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.127968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127881 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.127968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127931 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.127968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127931 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.128184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.127989 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.159832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.159794 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:05.187762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.187734 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j4x8\" (UniqueName: \"kubernetes.io/projected/ca05f496-f0d3-4c51-a11c-6874c7b37171-kube-api-access-9j4x8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.187911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.187852 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5nd\" (UniqueName: \"kubernetes.io/projected/39bcb846-ac58-4cc0-905a-be3408373098-kube-api-access-2d5nd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.188194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.188162 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4g4\" (UniqueName: \"kubernetes.io/projected/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-kube-api-access-zk4g4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.194026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.193995 5633 generic.go:356] "Generic (PLEG): container finished" podID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerID="8e5df8572a4ee22183c4018a349cf8191df48c381cd05b32c573775262b94af2" exitCode=0 Nov 01 20:06:05.194498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.194052 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" event={"ID":"ee343e4a-952a-44d4-8ca8-7f99975f1208","Type":"ContainerDied","Data":"8e5df8572a4ee22183c4018a349cf8191df48c381cd05b32c573775262b94af2"} Nov 01 20:06:05.215690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.215620 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:05.269030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.268936 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:05.487878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.487844 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:05.533050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.533007 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc"] Nov 01 20:06:05.633350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.633203 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds"] Nov 01 20:06:05.635878 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:05.635852 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bcb846_ac58_4cc0_905a_be3408373098.slice/crio-99755bb1f230f72f6bd0297553ee5e7a6544e4483e7f6bc3df506c9d44b98b17 WatchSource:0}: Error finding container 99755bb1f230f72f6bd0297553ee5e7a6544e4483e7f6bc3df506c9d44b98b17: Status 404 returned error can't find the container with id 99755bb1f230f72f6bd0297553ee5e7a6544e4483e7f6bc3df506c9d44b98b17 Nov 01 20:06:05.646546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.646498 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j"] Nov 01 20:06:05.651597 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:05.651562 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca05f496_f0d3_4c51_a11c_6874c7b37171.slice/crio-8ee80d38b851b11a32cb3b52049db14032649416d16caa12560901a33491400a WatchSource:0}: Error finding container 8ee80d38b851b11a32cb3b52049db14032649416d16caa12560901a33491400a: Status 404 returned error can't find the container with id 8ee80d38b851b11a32cb3b52049db14032649416d16caa12560901a33491400a Nov 01 20:06:05.860670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:05.860636 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss"] Nov 01 20:06:05.903219 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:05.903173 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09a5d3b_f64e_4dc5_9b30_feaccaecb373.slice/crio-71e2c6d6d509961edfce1a995b3e3a914b315caf0ff7139dd7209fb653b345de WatchSource:0}: Error finding container 71e2c6d6d509961edfce1a995b3e3a914b315caf0ff7139dd7209fb653b345de: Status 404 returned error can't find the container with id 71e2c6d6d509961edfce1a995b3e3a914b315caf0ff7139dd7209fb653b345de Nov 01 20:06:06.032850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.032816 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:06.201566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.201529 5633 generic.go:356] "Generic (PLEG): container finished" podID="39bcb846-ac58-4cc0-905a-be3408373098" containerID="8bf6409612b05bf14fdad1aa742eb465847fb1203ba12d5f4700c47895dd7bcd" exitCode=0 Nov 01 20:06:06.202074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.201631 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" event={"ID":"39bcb846-ac58-4cc0-905a-be3408373098","Type":"ContainerDied","Data":"8bf6409612b05bf14fdad1aa742eb465847fb1203ba12d5f4700c47895dd7bcd"} Nov 01 20:06:06.202074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.201695 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" event={"ID":"39bcb846-ac58-4cc0-905a-be3408373098","Type":"ContainerStarted","Data":"99755bb1f230f72f6bd0297553ee5e7a6544e4483e7f6bc3df506c9d44b98b17"} Nov 01 20:06:06.208069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.208030 5633 generic.go:356] "Generic (PLEG): container finished" podID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerID="7c8c11070334d5035bda65347edbab9b3ae8f43957e780dde26d11be920b691b" exitCode=0 Nov 01 20:06:06.208387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.208247 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" event={"ID":"4b2dfcef-ceb9-4518-96a3-c68b04071626","Type":"ContainerDied","Data":"7c8c11070334d5035bda65347edbab9b3ae8f43957e780dde26d11be920b691b"} Nov 01 20:06:06.208387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.208319 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" event={"ID":"4b2dfcef-ceb9-4518-96a3-c68b04071626","Type":"ContainerStarted","Data":"84844b42bca4124da79c3c15ddc0559800ff8113c2efdad5ef035a77d2bdb055"} Nov 01 20:06:06.210074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.210048 5633 generic.go:356] "Generic (PLEG): container finished" podID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerID="863cce2444364b11da899dc4efcaaf9ae37ad0cfb89f506b929ea4c60e12534b" exitCode=0 Nov 01 20:06:06.210172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.210076 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" event={"ID":"e09a5d3b-f64e-4dc5-9b30-feaccaecb373","Type":"ContainerDied","Data":"863cce2444364b11da899dc4efcaaf9ae37ad0cfb89f506b929ea4c60e12534b"} Nov 01 20:06:06.210172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.210119 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" event={"ID":"e09a5d3b-f64e-4dc5-9b30-feaccaecb373","Type":"ContainerStarted","Data":"71e2c6d6d509961edfce1a995b3e3a914b315caf0ff7139dd7209fb653b345de"} Nov 01 20:06:06.212429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.212409 5633 generic.go:356] "Generic (PLEG): container finished" podID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerID="805f417ab727c7dfeaedfa2538dc7fd1aee37ba22824a3b086c605a8b585d9b1" exitCode=0 Nov 01 20:06:06.212517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.212453 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" event={"ID":"ca05f496-f0d3-4c51-a11c-6874c7b37171","Type":"ContainerDied","Data":"805f417ab727c7dfeaedfa2538dc7fd1aee37ba22824a3b086c605a8b585d9b1"} Nov 01 20:06:06.212555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.212512 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" event={"ID":"ca05f496-f0d3-4c51-a11c-6874c7b37171","Type":"ContainerStarted","Data":"8ee80d38b851b11a32cb3b52049db14032649416d16caa12560901a33491400a"} Nov 01 20:06:06.223842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.223811 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:06.589045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.588995 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:06.828482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.828371 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:06.960245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:06.960206 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:08.228304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:08.228273 5633 generic.go:356] "Generic (PLEG): container finished" podID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerID="26bf3077a04abfc7048a8738e7d9db687e1c8bc3c03c4175f22c23079df66e0c" exitCode=0 Nov 01 20:06:08.228806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:08.228428 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" event={"ID":"255344a5-c362-4eda-8e42-84e6bdd791bb","Type":"ContainerDied","Data":"26bf3077a04abfc7048a8738e7d9db687e1c8bc3c03c4175f22c23079df66e0c"} Nov 01 20:06:08.230470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:08.230358 5633 generic.go:356] "Generic (PLEG): container finished" podID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerID="dc62ed0b5670d467ff58026c8b55dfa24248f6bd3ba247a7cf793a22d7ebdc85" exitCode=0 Nov 01 20:06:08.230470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:08.230437 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" event={"ID":"ee343e4a-952a-44d4-8ca8-7f99975f1208","Type":"ContainerDied","Data":"dc62ed0b5670d467ff58026c8b55dfa24248f6bd3ba247a7cf793a22d7ebdc85"} Nov 01 20:06:08.232448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:08.232426 5633 generic.go:356] "Generic (PLEG): container finished" podID="39bcb846-ac58-4cc0-905a-be3408373098" containerID="0312490f7c9f1a8c87ef66da56a9e1f972cc5d9560b202b5d28ee14a6be82ac6" exitCode=0 Nov 01 20:06:08.232547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:08.232503 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" event={"ID":"39bcb846-ac58-4cc0-905a-be3408373098","Type":"ContainerDied","Data":"0312490f7c9f1a8c87ef66da56a9e1f972cc5d9560b202b5d28ee14a6be82ac6"} Nov 01 20:06:09.241930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:09.241893 5633 generic.go:356] "Generic (PLEG): container finished" podID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerID="f9565998f4112671035a165f9df3ece257b3af02ffbf4e9f3ce20d8b1ec4c111" exitCode=0 Nov 01 20:06:09.241930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:09.241995 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" event={"ID":"4b2dfcef-ceb9-4518-96a3-c68b04071626","Type":"ContainerDied","Data":"f9565998f4112671035a165f9df3ece257b3af02ffbf4e9f3ce20d8b1ec4c111"} Nov 01 20:06:09.245487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:09.245453 5633 generic.go:356] "Generic (PLEG): container finished" podID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerID="20e308d316ca32cd965556bdd551e504b5b37a25603ee2a65d1016490b391dd5" exitCode=0 Nov 01 20:06:09.245626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:09.245537 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" event={"ID":"ca05f496-f0d3-4c51-a11c-6874c7b37171","Type":"ContainerDied","Data":"20e308d316ca32cd965556bdd551e504b5b37a25603ee2a65d1016490b391dd5"} Nov 01 20:06:10.255087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:10.255043 5633 generic.go:356] "Generic (PLEG): container finished" podID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerID="d57ce725fe88ae26b50534c802efd1f6ea73ba548f0236f316de277d75f59775" exitCode=0 Nov 01 20:06:10.255836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:10.255112 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" event={"ID":"e09a5d3b-f64e-4dc5-9b30-feaccaecb373","Type":"ContainerDied","Data":"d57ce725fe88ae26b50534c802efd1f6ea73ba548f0236f316de277d75f59775"} Nov 01 20:06:10.596648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:10.596607 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:16.177878 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:06:16.177805 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:06:18.206188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:18.205994 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:18.415869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:18.415826 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:21.340845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:21.340719 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" event={"ID":"39bcb846-ac58-4cc0-905a-be3408373098","Type":"ContainerStarted","Data":"ca51dd4a9dff3ea32add0b96c11a967af30a54dacdb6a0753c9a633212fb2f43"} Nov 01 20:06:21.459507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:21.459441 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" podStartSLOduration=3.579015956 podStartE2EDuration="17.45942106s" totalImagesPullingTime="13.880405104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:21.45760063 +0000 UTC m=+189.131859886" watchObservedRunningTime="2026-11-01 20:06:21.45942106 +0000 UTC m=+189.133680316" Nov 01 20:06:21.496359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:21.496300 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:21.649036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:21.648940 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:22.349191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:22.349154 5633 generic.go:356] "Generic (PLEG): container finished" podID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerID="7e75d381636d23087a87190548e9a532b73c48a999c53e65e3d3c78bae9a0756" exitCode=0 Nov 01 20:06:22.349710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:22.349686 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" event={"ID":"255344a5-c362-4eda-8e42-84e6bdd791bb","Type":"ContainerDied","Data":"7e75d381636d23087a87190548e9a532b73c48a999c53e65e3d3c78bae9a0756"} Nov 01 20:06:22.352591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:22.352559 5633 generic.go:356] "Generic (PLEG): container finished" podID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerID="77bff51f1fa8d8cd2883774e947df68b027ea494e5e844b6ca27604a33668c27" exitCode=0 Nov 01 20:06:22.352800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:22.352772 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" event={"ID":"ee343e4a-952a-44d4-8ca8-7f99975f1208","Type":"ContainerDied","Data":"77bff51f1fa8d8cd2883774e947df68b027ea494e5e844b6ca27604a33668c27"} Nov 01 20:06:22.356449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:22.356421 5633 generic.go:356] "Generic (PLEG): container finished" podID="39bcb846-ac58-4cc0-905a-be3408373098" containerID="ca51dd4a9dff3ea32add0b96c11a967af30a54dacdb6a0753c9a633212fb2f43" exitCode=0 Nov 01 20:06:22.356595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:22.356475 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" event={"ID":"39bcb846-ac58-4cc0-905a-be3408373098","Type":"ContainerDied","Data":"ca51dd4a9dff3ea32add0b96c11a967af30a54dacdb6a0753c9a633212fb2f43"} Nov 01 20:06:23.363345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.363301 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" event={"ID":"4b2dfcef-ceb9-4518-96a3-c68b04071626","Type":"ContainerStarted","Data":"613f505a5fded512ee3f9c604fcc76abf1f75facb8d93e5698320522d031646c"} Nov 01 20:06:23.365663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.365626 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" event={"ID":"ca05f496-f0d3-4c51-a11c-6874c7b37171","Type":"ContainerStarted","Data":"f76ceb708343d16284eda45840b90a65802454604c5768c2e62661af2e4fb909"} Nov 01 20:06:23.481184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.481127 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" podStartSLOduration=3.439801493 podStartE2EDuration="19.481105577s" totalImagesPullingTime="16.041304084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:23.479694646 +0000 UTC m=+191.153953903" watchObservedRunningTime="2026-11-01 20:06:23.481105577 +0000 UTC m=+191.155364835" Nov 01 20:06:23.526028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.525998 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:06:23.562055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.562024 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:23.578079 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.578052 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:06:23.594991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.594931 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" podStartSLOduration=3.4008110670000002 podStartE2EDuration="19.594916469s" totalImagesPullingTime="16.194105402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:23.594324896 +0000 UTC m=+191.268584150" watchObservedRunningTime="2026-11-01 20:06:23.594916469 +0000 UTC m=+191.269175760" Nov 01 20:06:23.616624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616593 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-util\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-util\") pod \"39bcb846-ac58-4cc0-905a-be3408373098\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " Nov 01 20:06:23.616770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616633 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-bundle\") pod \"255344a5-c362-4eda-8e42-84e6bdd791bb\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " Nov 01 20:06:23.616770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616679 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-bundle\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-bundle\") pod \"39bcb846-ac58-4cc0-905a-be3408373098\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " Nov 01 20:06:23.616770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616743 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-util\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-util\") pod \"ee343e4a-952a-44d4-8ca8-7f99975f1208\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " Nov 01 20:06:23.616911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616805 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bcb846-ac58-4cc0-905a-be3408373098-kube-api-access-2d5nd\" (UniqueName: \"kubernetes.io/projected/39bcb846-ac58-4cc0-905a-be3408373098-kube-api-access-2d5nd\") pod \"39bcb846-ac58-4cc0-905a-be3408373098\" (UID: \"39bcb846-ac58-4cc0-905a-be3408373098\") " Nov 01 20:06:23.616911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616838 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-bundle\") pod \"ee343e4a-952a-44d4-8ca8-7f99975f1208\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " Nov 01 20:06:23.616911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616878 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255344a5-c362-4eda-8e42-84e6bdd791bb-kube-api-access-bpvjk\" (UniqueName: \"kubernetes.io/projected/255344a5-c362-4eda-8e42-84e6bdd791bb-kube-api-access-bpvjk\") pod \"255344a5-c362-4eda-8e42-84e6bdd791bb\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " Nov 01 20:06:23.617069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.616991 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-util\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-util\") pod \"255344a5-c362-4eda-8e42-84e6bdd791bb\" (UID: \"255344a5-c362-4eda-8e42-84e6bdd791bb\") " Nov 01 20:06:23.617069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.617039 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee343e4a-952a-44d4-8ca8-7f99975f1208-kube-api-access-r9btn\" (UniqueName: \"kubernetes.io/projected/ee343e4a-952a-44d4-8ca8-7f99975f1208-kube-api-access-r9btn\") pod \"ee343e4a-952a-44d4-8ca8-7f99975f1208\" (UID: \"ee343e4a-952a-44d4-8ca8-7f99975f1208\") " Nov 01 20:06:23.617291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.617260 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-bundle" pod "255344a5-c362-4eda-8e42-84e6bdd791bb" (UID: "255344a5-c362-4eda-8e42-84e6bdd791bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:23.617400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.617319 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-bundle" pod "39bcb846-ac58-4cc0-905a-be3408373098" (UID: "39bcb846-ac58-4cc0-905a-be3408373098"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:23.617464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.617401 5633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.617512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.617466 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-bundle" pod "ee343e4a-952a-44d4-8ca8-7f99975f1208" (UID: "ee343e4a-952a-44d4-8ca8-7f99975f1208"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:23.620052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.620024 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee343e4a-952a-44d4-8ca8-7f99975f1208-kube-api-access-r9btn" pod "ee343e4a-952a-44d4-8ca8-7f99975f1208" (UID: "ee343e4a-952a-44d4-8ca8-7f99975f1208"). InnerVolumeSpecName "kube-api-access-r9btn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:06:23.620052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.620037 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bcb846-ac58-4cc0-905a-be3408373098-kube-api-access-2d5nd" pod "39bcb846-ac58-4cc0-905a-be3408373098" (UID: "39bcb846-ac58-4cc0-905a-be3408373098"). InnerVolumeSpecName "kube-api-access-2d5nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:06:23.620247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.620028 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255344a5-c362-4eda-8e42-84e6bdd791bb-kube-api-access-bpvjk" pod "255344a5-c362-4eda-8e42-84e6bdd791bb" (UID: "255344a5-c362-4eda-8e42-84e6bdd791bb"). InnerVolumeSpecName "kube-api-access-bpvjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:06:23.621712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.621691 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-util" pod "ee343e4a-952a-44d4-8ca8-7f99975f1208" (UID: "ee343e4a-952a-44d4-8ca8-7f99975f1208"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:23.622785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.622761 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-util" pod "255344a5-c362-4eda-8e42-84e6bdd791bb" (UID: "255344a5-c362-4eda-8e42-84e6bdd791bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:23.623028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.623010 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-util" pod "39bcb846-ac58-4cc0-905a-be3408373098" (UID: "39bcb846-ac58-4cc0-905a-be3408373098"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:23.718728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718694 5633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-util\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.718728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718724 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5nd\" (UniqueName: \"kubernetes.io/projected/39bcb846-ac58-4cc0-905a-be3408373098-kube-api-access-2d5nd\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.718728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718735 5633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee343e4a-952a-44d4-8ca8-7f99975f1208-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.718947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718744 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpvjk\" (UniqueName: \"kubernetes.io/projected/255344a5-c362-4eda-8e42-84e6bdd791bb-kube-api-access-bpvjk\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.718947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718774 5633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255344a5-c362-4eda-8e42-84e6bdd791bb-util\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.718947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718784 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9btn\" (UniqueName: \"kubernetes.io/projected/ee343e4a-952a-44d4-8ca8-7f99975f1208-kube-api-access-r9btn\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.718947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718791 5633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-util\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:23.718947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:23.718800 5633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39bcb846-ac58-4cc0-905a-be3408373098-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:24.371287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.371256 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" Nov 01 20:06:24.371287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.371273 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448bsjd" event={"ID":"255344a5-c362-4eda-8e42-84e6bdd791bb","Type":"ContainerDied","Data":"429d7a60c628fa7d490736e12ff67efa830594f494c9db2013b8abe0127e3cac"} Nov 01 20:06:24.371827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.371323 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="429d7a60c628fa7d490736e12ff67efa830594f494c9db2013b8abe0127e3cac" Nov 01 20:06:24.373229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.373189 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" event={"ID":"ee343e4a-952a-44d4-8ca8-7f99975f1208","Type":"ContainerDied","Data":"378a0430eb6dda6ba25b18025894b4e0b4ef2ddf4e26026c127d0d8c738f6fba"} Nov 01 20:06:24.373229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.373222 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7m7" Nov 01 20:06:24.373455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.373241 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378a0430eb6dda6ba25b18025894b4e0b4ef2ddf4e26026c127d0d8c738f6fba" Nov 01 20:06:24.375035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.375012 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" event={"ID":"39bcb846-ac58-4cc0-905a-be3408373098","Type":"ContainerDied","Data":"99755bb1f230f72f6bd0297553ee5e7a6544e4483e7f6bc3df506c9d44b98b17"} Nov 01 20:06:24.375149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.375052 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99755bb1f230f72f6bd0297553ee5e7a6544e4483e7f6bc3df506c9d44b98b17" Nov 01 20:06:24.375149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.375025 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78rzds" Nov 01 20:06:24.377491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.377465 5633 generic.go:356] "Generic (PLEG): container finished" podID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerID="613f505a5fded512ee3f9c604fcc76abf1f75facb8d93e5698320522d031646c" exitCode=0 Nov 01 20:06:24.377661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.377543 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" event={"ID":"4b2dfcef-ceb9-4518-96a3-c68b04071626","Type":"ContainerDied","Data":"613f505a5fded512ee3f9c604fcc76abf1f75facb8d93e5698320522d031646c"} Nov 01 20:06:24.379891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.379863 5633 generic.go:356] "Generic (PLEG): container finished" podID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerID="5b46cbb6952a6ffcf33220848c7c13b4fd1f2ee3d575977d8f30c9d8204ad3d4" exitCode=0 Nov 01 20:06:24.380021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.379946 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" event={"ID":"e09a5d3b-f64e-4dc5-9b30-feaccaecb373","Type":"ContainerDied","Data":"5b46cbb6952a6ffcf33220848c7c13b4fd1f2ee3d575977d8f30c9d8204ad3d4"} Nov 01 20:06:24.382491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.382465 5633 generic.go:356] "Generic (PLEG): container finished" podID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerID="f76ceb708343d16284eda45840b90a65802454604c5768c2e62661af2e4fb909" exitCode=0 Nov 01 20:06:24.382591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:24.382505 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" event={"ID":"ca05f496-f0d3-4c51-a11c-6874c7b37171","Type":"ContainerDied","Data":"f76ceb708343d16284eda45840b90a65802454604c5768c2e62661af2e4fb909"} Nov 01 20:06:25.537873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.537843 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:25.575905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.575879 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:25.578532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.578511 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:25.635638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635604 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-bundle\") pod \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " Nov 01 20:06:25.635828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635670 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-bundle\") pod \"4b2dfcef-ceb9-4518-96a3-c68b04071626\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " Nov 01 20:06:25.635828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635699 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-util\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-util\") pod \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " Nov 01 20:06:25.635828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635746 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-util\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-util\") pod \"4b2dfcef-ceb9-4518-96a3-c68b04071626\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " Nov 01 20:06:25.635828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635775 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2dfcef-ceb9-4518-96a3-c68b04071626-kube-api-access-dczct\" (UniqueName: \"kubernetes.io/projected/4b2dfcef-ceb9-4518-96a3-c68b04071626-kube-api-access-dczct\") pod \"4b2dfcef-ceb9-4518-96a3-c68b04071626\" (UID: \"4b2dfcef-ceb9-4518-96a3-c68b04071626\") " Nov 01 20:06:25.635828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635826 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca05f496-f0d3-4c51-a11c-6874c7b37171-kube-api-access-9j4x8\" (UniqueName: \"kubernetes.io/projected/ca05f496-f0d3-4c51-a11c-6874c7b37171-kube-api-access-9j4x8\") pod \"ca05f496-f0d3-4c51-a11c-6874c7b37171\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " Nov 01 20:06:25.636090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635873 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-kube-api-access-zk4g4\" (UniqueName: \"kubernetes.io/projected/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-kube-api-access-zk4g4\") pod \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\" (UID: \"e09a5d3b-f64e-4dc5-9b30-feaccaecb373\") " Nov 01 20:06:25.636090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635901 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-util\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-util\") pod \"ca05f496-f0d3-4c51-a11c-6874c7b37171\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " Nov 01 20:06:25.636090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.635960 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-bundle\") pod \"ca05f496-f0d3-4c51-a11c-6874c7b37171\" (UID: \"ca05f496-f0d3-4c51-a11c-6874c7b37171\") " Nov 01 20:06:25.636236 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.636126 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-bundle" pod "e09a5d3b-f64e-4dc5-9b30-feaccaecb373" (UID: "e09a5d3b-f64e-4dc5-9b30-feaccaecb373"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:25.636291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.636245 5633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.636839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.636808 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-bundle" pod "4b2dfcef-ceb9-4518-96a3-c68b04071626" (UID: "4b2dfcef-ceb9-4518-96a3-c68b04071626"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:25.636959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.636867 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-bundle" pod "ca05f496-f0d3-4c51-a11c-6874c7b37171" (UID: "ca05f496-f0d3-4c51-a11c-6874c7b37171"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:25.638991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.638929 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-kube-api-access-zk4g4" pod "e09a5d3b-f64e-4dc5-9b30-feaccaecb373" (UID: "e09a5d3b-f64e-4dc5-9b30-feaccaecb373"). InnerVolumeSpecName "kube-api-access-zk4g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:06:25.639090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.639019 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2dfcef-ceb9-4518-96a3-c68b04071626-kube-api-access-dczct" pod "4b2dfcef-ceb9-4518-96a3-c68b04071626" (UID: "4b2dfcef-ceb9-4518-96a3-c68b04071626"). InnerVolumeSpecName "kube-api-access-dczct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:06:25.640685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.640659 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca05f496-f0d3-4c51-a11c-6874c7b37171-kube-api-access-9j4x8" pod "ca05f496-f0d3-4c51-a11c-6874c7b37171" (UID: "ca05f496-f0d3-4c51-a11c-6874c7b37171"). InnerVolumeSpecName "kube-api-access-9j4x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:06:25.641852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.641830 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-util" pod "4b2dfcef-ceb9-4518-96a3-c68b04071626" (UID: "4b2dfcef-ceb9-4518-96a3-c68b04071626"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:25.642166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.642146 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-util" pod "e09a5d3b-f64e-4dc5-9b30-feaccaecb373" (UID: "e09a5d3b-f64e-4dc5-9b30-feaccaecb373"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:25.643504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.643485 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-util" pod "ca05f496-f0d3-4c51-a11c-6874c7b37171" (UID: "ca05f496-f0d3-4c51-a11c-6874c7b37171"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:06:25.736815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736775 5633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-util\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.736815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736805 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczct\" (UniqueName: \"kubernetes.io/projected/4b2dfcef-ceb9-4518-96a3-c68b04071626-kube-api-access-dczct\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.736815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736816 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j4x8\" (UniqueName: \"kubernetes.io/projected/ca05f496-f0d3-4c51-a11c-6874c7b37171-kube-api-access-9j4x8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.737065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736827 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4g4\" (UniqueName: \"kubernetes.io/projected/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-kube-api-access-zk4g4\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.737065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736836 5633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-util\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.737065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736845 5633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca05f496-f0d3-4c51-a11c-6874c7b37171-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.737065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736853 5633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b2dfcef-ceb9-4518-96a3-c68b04071626-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:25.737065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:25.736860 5633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09a5d3b-f64e-4dc5-9b30-feaccaecb373-util\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:06:26.393460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.393422 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" event={"ID":"4b2dfcef-ceb9-4518-96a3-c68b04071626","Type":"ContainerDied","Data":"84844b42bca4124da79c3c15ddc0559800ff8113c2efdad5ef035a77d2bdb055"} Nov 01 20:06:26.393640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.393494 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84844b42bca4124da79c3c15ddc0559800ff8113c2efdad5ef035a77d2bdb055" Nov 01 20:06:26.393640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.393438 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awn9qc" Nov 01 20:06:26.395549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.395519 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" event={"ID":"e09a5d3b-f64e-4dc5-9b30-feaccaecb373","Type":"ContainerDied","Data":"71e2c6d6d509961edfce1a995b3e3a914b315caf0ff7139dd7209fb653b345de"} Nov 01 20:06:26.395689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.395550 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w9fss" Nov 01 20:06:26.395689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.395564 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e2c6d6d509961edfce1a995b3e3a914b315caf0ff7139dd7209fb653b345de" Nov 01 20:06:26.397546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.397519 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" event={"ID":"ca05f496-f0d3-4c51-a11c-6874c7b37171","Type":"ContainerDied","Data":"8ee80d38b851b11a32cb3b52049db14032649416d16caa12560901a33491400a"} Nov 01 20:06:26.397686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.397557 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ee80d38b851b11a32cb3b52049db14032649416d16caa12560901a33491400a" Nov 01 20:06:26.397686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.397561 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cws29j" Nov 01 20:06:26.827987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:26.827950 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:30.917325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:30.917290 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:33.390683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.390638 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns"] Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391108 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerName="util" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391124 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerName="util" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391134 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerName="pull" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391138 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerName="pull" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391147 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerName="extract" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391153 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerName="extract" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391161 5633 cpu_manager.go:414] "containerMap: removing container" podUID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerName="util" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391166 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerName="util" Nov 01 20:06:33.391170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391172 5633 cpu_manager.go:414] "containerMap: removing container" podUID="39bcb846-ac58-4cc0-905a-be3408373098" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391177 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bcb846-ac58-4cc0-905a-be3408373098" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391183 5633 cpu_manager.go:414] "containerMap: removing container" podUID="39bcb846-ac58-4cc0-905a-be3408373098" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391188 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bcb846-ac58-4cc0-905a-be3408373098" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391198 5633 cpu_manager.go:414] "containerMap: removing container" podUID="39bcb846-ac58-4cc0-905a-be3408373098" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391202 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bcb846-ac58-4cc0-905a-be3408373098" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391207 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391211 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391221 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391225 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391234 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391238 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391244 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391248 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391254 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391258 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391274 5633 cpu_manager.go:414] "containerMap: removing container" podUID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391278 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391283 5633 cpu_manager.go:414] "containerMap: removing container" podUID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391288 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391295 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391299 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerName="pull" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391307 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391311 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391316 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391320 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerName="util" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391341 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391348 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391402 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09a5d3b-f64e-4dc5-9b30-feaccaecb373" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391412 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca05f496-f0d3-4c51-a11c-6874c7b37171" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391418 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="255344a5-c362-4eda-8e42-84e6bdd791bb" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391423 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2dfcef-ceb9-4518-96a3-c68b04071626" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391428 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee343e4a-952a-44d4-8ca8-7f99975f1208" containerName="extract" Nov 01 20:06:33.391525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.391433 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bcb846-ac58-4cc0-905a-be3408373098" containerName="extract" Nov 01 20:06:33.393824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.393806 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 20:06:33.501263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.501217 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 20:06:33.501263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.501248 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 20:06:33.501263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.501234 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 20:06:33.501263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.501217 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4jhrn\"" Nov 01 20:06:33.502460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.502438 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns"] Nov 01 20:06:33.504784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.504760 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nww2\" (UniqueName: \"kubernetes.io/projected/e704eaa9-50e3-4234-9357-f374ad06a72a-kube-api-access-5nww2\") pod \"dns-operator-controller-manager-8557cbdb84-mlhns\" (UID: \"e704eaa9-50e3-4234-9357-f374ad06a72a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 20:06:33.605785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.605749 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nww2\" (UniqueName: \"kubernetes.io/projected/e704eaa9-50e3-4234-9357-f374ad06a72a-kube-api-access-5nww2\") pod \"dns-operator-controller-manager-8557cbdb84-mlhns\" (UID: \"e704eaa9-50e3-4234-9357-f374ad06a72a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 20:06:33.668451 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.668366 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nww2\" (UniqueName: \"kubernetes.io/projected/e704eaa9-50e3-4234-9357-f374ad06a72a-kube-api-access-5nww2\") pod \"dns-operator-controller-manager-8557cbdb84-mlhns\" (UID: \"e704eaa9-50e3-4234-9357-f374ad06a72a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 20:06:33.705199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:33.705157 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 20:06:34.129601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:34.129562 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns"] Nov 01 20:06:34.133023 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:34.132989 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode704eaa9_50e3_4234_9357_f374ad06a72a.slice/crio-eeacabea78c33296c49dfec03bc2f05b55c9623ba0c196265a4d6d1afe9dec8e WatchSource:0}: Error finding container eeacabea78c33296c49dfec03bc2f05b55c9623ba0c196265a4d6d1afe9dec8e: Status 404 returned error can't find the container with id eeacabea78c33296c49dfec03bc2f05b55c9623ba0c196265a4d6d1afe9dec8e Nov 01 20:06:34.430470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:34.430389 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" event={"ID":"e704eaa9-50e3-4234-9357-f374ad06a72a","Type":"ContainerStarted","Data":"eeacabea78c33296c49dfec03bc2f05b55c9623ba0c196265a4d6d1afe9dec8e"} Nov 01 20:06:34.546322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:34.546279 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:36.575621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.575576 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:36.790291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.790260 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s"] Nov 01 20:06:36.792561 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.792538 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" Nov 01 20:06:36.827197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.827122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:36.837142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.837114 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/a8e46f26-efa8-4823-b684-b805cce7bf88-kube-api-access-xnj58\") pod \"limitador-operator-controller-manager-5778bb4449-wdb8s\" (UID: \"a8e46f26-efa8-4823-b684-b805cce7bf88\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" Nov 01 20:06:36.846040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.845995 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-w2kmv\"" Nov 01 20:06:36.897171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.897134 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s"] Nov 01 20:06:36.938417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.938380 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/a8e46f26-efa8-4823-b684-b805cce7bf88-kube-api-access-xnj58\") pod \"limitador-operator-controller-manager-5778bb4449-wdb8s\" (UID: \"a8e46f26-efa8-4823-b684-b805cce7bf88\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" Nov 01 20:06:36.994795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:36.994763 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnj58\" (UniqueName: \"kubernetes.io/projected/a8e46f26-efa8-4823-b684-b805cce7bf88-kube-api-access-xnj58\") pod \"limitador-operator-controller-manager-5778bb4449-wdb8s\" (UID: \"a8e46f26-efa8-4823-b684-b805cce7bf88\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" Nov 01 20:06:37.108266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:37.108181 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" Nov 01 20:06:37.486921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:37.486882 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s"] Nov 01 20:06:37.490408 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:37.490379 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e46f26_efa8_4823_b684_b805cce7bf88.slice/crio-d213644852bce81ea51b247a9adbbd839b498723920c9ce0d4938eb7da991605 WatchSource:0}: Error finding container d213644852bce81ea51b247a9adbbd839b498723920c9ce0d4938eb7da991605: Status 404 returned error can't find the container with id d213644852bce81ea51b247a9adbbd839b498723920c9ce0d4938eb7da991605 Nov 01 20:06:37.699740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:37.699695 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:37.926113 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:37.926030 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:38.015267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.015227 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7"] Nov 01 20:06:38.017991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.017963 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.070088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.070047 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 20:06:38.070088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.070051 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 20:06:38.070088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.070063 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 20:06:38.070403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.070183 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 20:06:38.070403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.070231 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 20:06:38.070403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.070250 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sx9cg\"" Nov 01 20:06:38.129143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.129100 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:38.150444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.150413 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a0bb745-0218-4afc-ab40-562a2f70271b-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.150584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.150450 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.150584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.150571 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.150662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.150610 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a0bb745-0218-4afc-ab40-562a2f70271b-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.150699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.150680 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.150768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.150750 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5crq\" (UniqueName: \"kubernetes.io/projected/0a0bb745-0218-4afc-ab40-562a2f70271b-kube-api-access-q5crq\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.251191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.247042 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7"] Nov 01 20:06:38.251650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.251618 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a0bb745-0218-4afc-ab40-562a2f70271b-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.251770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.251669 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.251827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.251768 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.251889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.251831 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a0bb745-0218-4afc-ab40-562a2f70271b-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.251889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.251882 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.251982 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.251909 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5crq\" (UniqueName: \"kubernetes.io/projected/0a0bb745-0218-4afc-ab40-562a2f70271b-kube-api-access-q5crq\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.252696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.252643 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a0bb745-0218-4afc-ab40-562a2f70271b-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.254650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.254622 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a0bb745-0218-4afc-ab40-562a2f70271b-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.254934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.254809 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.254934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.254908 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.254934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.254916 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a0bb745-0218-4afc-ab40-562a2f70271b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.275865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.275832 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:38.309057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.309024 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5crq\" (UniqueName: \"kubernetes.io/projected/0a0bb745-0218-4afc-ab40-562a2f70271b-kube-api-access-q5crq\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7\" (UID: \"0a0bb745-0218-4afc-ab40-562a2f70271b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.334369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.334319 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:38.435268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.435227 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:38.453435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.453223 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" event={"ID":"e704eaa9-50e3-4234-9357-f374ad06a72a","Type":"ContainerStarted","Data":"6643642df36908f53ff88429637cda6fa766043abbd11f2e486207f2530aaf3a"} Nov 01 20:06:38.453670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.453352 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 20:06:38.454819 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.454793 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" event={"ID":"a8e46f26-efa8-4823-b684-b805cce7bf88","Type":"ContainerStarted","Data":"d213644852bce81ea51b247a9adbbd839b498723920c9ce0d4938eb7da991605"} Nov 01 20:06:38.568823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.568713 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" podStartSLOduration=1.601024512 podStartE2EDuration="5.568695599s" totalImagesPullingTime="3.967671087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:38.567709603 +0000 UTC m=+206.241968865" watchObservedRunningTime="2026-11-01 20:06:38.568695599 +0000 UTC m=+206.242954861" Nov 01 20:06:38.576863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.576828 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:38.722941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.722900 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:38.832835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.832812 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7"] Nov 01 20:06:38.834587 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:38.834561 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0bb745_0218_4afc_ab40_562a2f70271b.slice/crio-936681142e8db187ab0cce8edb25c0f1defb06eaa99069dcef32505564f20453 WatchSource:0}: Error finding container 936681142e8db187ab0cce8edb25c0f1defb06eaa99069dcef32505564f20453: Status 404 returned error can't find the container with id 936681142e8db187ab0cce8edb25c0f1defb06eaa99069dcef32505564f20453 Nov 01 20:06:38.909983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:38.909943 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:39.045072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:39.045031 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:39.461115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:39.461066 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" event={"ID":"0a0bb745-0218-4afc-ab40-562a2f70271b","Type":"ContainerStarted","Data":"936681142e8db187ab0cce8edb25c0f1defb06eaa99069dcef32505564f20453"} Nov 01 20:06:39.903204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:39.903124 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:40.469698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:40.469208 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" event={"ID":"a8e46f26-efa8-4823-b684-b805cce7bf88","Type":"ContainerStarted","Data":"4c6ff730487a503624cb8bdde3e4ef727f21541d221092ff0beceda137007849"} Nov 01 20:06:40.469698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:40.469374 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" Nov 01 20:06:40.586636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:40.586528 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" podStartSLOduration=1.784660004 podStartE2EDuration="4.586505633s" totalImagesPullingTime="2.801845629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:40.585468918 +0000 UTC m=+208.259728174" watchObservedRunningTime="2026-11-01 20:06:40.586505633 +0000 UTC m=+208.260764890" Nov 01 20:06:42.704562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:42.704463 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:43.387300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:43.387214 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:43.484560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:43.484525 5633 generic.go:356] "Generic (PLEG): container finished" podID="0a0bb745-0218-4afc-ab40-562a2f70271b" containerID="996fbce9fa3566bd90db7cf6cf24008f8ed2b3ae2b5dd941bac949707a068eac" exitCode=0 Nov 01 20:06:43.484735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:43.484610 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" event={"ID":"0a0bb745-0218-4afc-ab40-562a2f70271b","Type":"ContainerDied","Data":"996fbce9fa3566bd90db7cf6cf24008f8ed2b3ae2b5dd941bac949707a068eac"} Nov 01 20:06:43.557855 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:43.557813 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:44.491965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:44.491923 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" event={"ID":"0a0bb745-0218-4afc-ab40-562a2f70271b","Type":"ContainerStarted","Data":"1b2c158609dff274d1d5035d1f6229d98799c18cc643fe5573e9891441fee08f"} Nov 01 20:06:44.492431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:44.492040 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:44.706377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:44.706273 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:44.724028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:44.723969 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" podStartSLOduration=2.748220343 podStartE2EDuration="7.723955111s" totalImagesPullingTime="4.476581346s" totalInitContainerRuntime="499.153422ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:44.674701275 +0000 UTC m=+212.348960535" watchObservedRunningTime="2026-11-01 20:06:44.723955111 +0000 UTC m=+212.398214357" Nov 01 20:06:45.982703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:45.982670 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:46.135901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:46.135865 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:46.157043 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:06:46.157002 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:06:46.339944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:46.339849 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:47.728034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:47.727996 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:49.300222 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:49.300185 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:49.463435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:49.463391 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 20:06:49.784985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:49.784951 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:50.063391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:50.063281 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:50.206083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:50.206045 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:50.388962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:50.388857 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:50.854975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:50.854943 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:50.978869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:50.978840 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:51.303324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.303284 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz"] Nov 01 20:06:51.315016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.314984 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:51.378137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.378072 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz"] Nov 01 20:06:51.425384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.425320 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 20:06:51.425384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.425392 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8nknn\"" Nov 01 20:06:51.425796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.425412 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 20:06:51.476580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.476550 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wdb8s" Nov 01 20:06:51.479418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.479393 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6981ce03-b7ca-4213-991c-379c3f82bbc6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nqzfz\" (UID: \"6981ce03-b7ca-4213-991c-379c3f82bbc6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:51.479583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.479443 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp82\" (UniqueName: \"kubernetes.io/projected/6981ce03-b7ca-4213-991c-379c3f82bbc6-kube-api-access-qnp82\") pod \"cert-manager-webhook-5dbbbf75cc-nqzfz\" (UID: \"6981ce03-b7ca-4213-991c-379c3f82bbc6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:51.580930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.580832 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6981ce03-b7ca-4213-991c-379c3f82bbc6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nqzfz\" (UID: \"6981ce03-b7ca-4213-991c-379c3f82bbc6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:51.580930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.580885 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp82\" (UniqueName: \"kubernetes.io/projected/6981ce03-b7ca-4213-991c-379c3f82bbc6-kube-api-access-qnp82\") pod \"cert-manager-webhook-5dbbbf75cc-nqzfz\" (UID: \"6981ce03-b7ca-4213-991c-379c3f82bbc6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:51.651565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.651528 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6981ce03-b7ca-4213-991c-379c3f82bbc6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nqzfz\" (UID: \"6981ce03-b7ca-4213-991c-379c3f82bbc6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:51.659867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.659836 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp82\" (UniqueName: \"kubernetes.io/projected/6981ce03-b7ca-4213-991c-379c3f82bbc6-kube-api-access-qnp82\") pod \"cert-manager-webhook-5dbbbf75cc-nqzfz\" (UID: \"6981ce03-b7ca-4213-991c-379c3f82bbc6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:51.945692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:51.945595 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:52.141473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.141437 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:52.284415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.284377 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:52.330660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.330628 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz"] Nov 01 20:06:52.333176 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:52.333146 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6981ce03_b7ca_4213_991c_379c3f82bbc6.slice/crio-a6c149e325fe23e138a7701e2c58df6bf3d769e016dd7760c2e3c2d2cb5e664e WatchSource:0}: Error finding container a6c149e325fe23e138a7701e2c58df6bf3d769e016dd7760c2e3c2d2cb5e664e: Status 404 returned error can't find the container with id a6c149e325fe23e138a7701e2c58df6bf3d769e016dd7760c2e3c2d2cb5e664e Nov 01 20:06:52.434762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.434722 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:52.528600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.528562 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" event={"ID":"6981ce03-b7ca-4213-991c-379c3f82bbc6","Type":"ContainerStarted","Data":"a6c149e325fe23e138a7701e2c58df6bf3d769e016dd7760c2e3c2d2cb5e664e"} Nov 01 20:06:52.890812 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.890778 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:52.936890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.936854 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9"] Nov 01 20:06:52.956464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:52.956436 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.003733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.003694 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9"] Nov 01 20:06:53.051544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.051506 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-45cmh\"" Nov 01 20:06:53.093037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.092994 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldpz\" (UniqueName: \"kubernetes.io/projected/b68f6371-dc7c-440c-8fda-13b8388e7221-kube-api-access-rldpz\") pod \"cert-manager-cainjector-5c57bbd4fc-sqxc9\" (UID: \"b68f6371-dc7c-440c-8fda-13b8388e7221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.093282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.093118 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b68f6371-dc7c-440c-8fda-13b8388e7221-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sqxc9\" (UID: \"b68f6371-dc7c-440c-8fda-13b8388e7221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.194145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.194047 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b68f6371-dc7c-440c-8fda-13b8388e7221-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sqxc9\" (UID: \"b68f6371-dc7c-440c-8fda-13b8388e7221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.194145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.194109 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldpz\" (UniqueName: \"kubernetes.io/projected/b68f6371-dc7c-440c-8fda-13b8388e7221-kube-api-access-rldpz\") pod \"cert-manager-cainjector-5c57bbd4fc-sqxc9\" (UID: \"b68f6371-dc7c-440c-8fda-13b8388e7221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.253400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.253367 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b68f6371-dc7c-440c-8fda-13b8388e7221-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sqxc9\" (UID: \"b68f6371-dc7c-440c-8fda-13b8388e7221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.254420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.254392 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldpz\" (UniqueName: \"kubernetes.io/projected/b68f6371-dc7c-440c-8fda-13b8388e7221-kube-api-access-rldpz\") pod \"cert-manager-cainjector-5c57bbd4fc-sqxc9\" (UID: \"b68f6371-dc7c-440c-8fda-13b8388e7221\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.276845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.276812 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" Nov 01 20:06:53.658317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:53.658224 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9"] Nov 01 20:06:53.660880 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:06:53.660847 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb68f6371_dc7c_440c_8fda_13b8388e7221.slice/crio-e3e879531f0bec66ea03933ae80700a08b455dec1aa2c2ea72e7bb38e51d3d0b WatchSource:0}: Error finding container e3e879531f0bec66ea03933ae80700a08b455dec1aa2c2ea72e7bb38e51d3d0b: Status 404 returned error can't find the container with id e3e879531f0bec66ea03933ae80700a08b455dec1aa2c2ea72e7bb38e51d3d0b Nov 01 20:06:54.541852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:54.541811 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" event={"ID":"b68f6371-dc7c-440c-8fda-13b8388e7221","Type":"ContainerStarted","Data":"e3e879531f0bec66ea03933ae80700a08b455dec1aa2c2ea72e7bb38e51d3d0b"} Nov 01 20:06:55.500470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:55.500435 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-jg4b7" Nov 01 20:06:55.913805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:55.913718 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:56.081983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:56.081222 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:56.211287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:56.211250 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:58.567868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:58.567829 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" event={"ID":"6981ce03-b7ca-4213-991c-379c3f82bbc6","Type":"ContainerStarted","Data":"aaa66124bdc97a80a89bc107317bba10bebf8aafae1deb90ea73d0f5ca3a2694"} Nov 01 20:06:58.568319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:58.567883 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:06:58.569535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:58.569507 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" event={"ID":"b68f6371-dc7c-440c-8fda-13b8388e7221","Type":"ContainerStarted","Data":"61a13e941f54f597b6a1224ab83b5b6d3ef8806b5d77eade2950ad72a317b5a4"} Nov 01 20:06:58.681785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:58.681645 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" podStartSLOduration=2.542913517 podStartE2EDuration="7.681625586s" totalImagesPullingTime="5.138712069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:58.681302744 +0000 UTC m=+226.355561997" watchObservedRunningTime="2026-11-01 20:06:58.681625586 +0000 UTC m=+226.355884842" Nov 01 20:06:58.796390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:58.796320 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" podStartSLOduration=2.450334539 podStartE2EDuration="6.796300178s" totalImagesPullingTime="4.345965639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:58.795736892 +0000 UTC m=+226.469996148" watchObservedRunningTime="2026-11-01 20:06:58.796300178 +0000 UTC m=+226.470559435" Nov 01 20:06:59.589137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:59.589097 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:06:59.831169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:06:59.831129 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:00.191161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:00.191120 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:01.363318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.363284 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:01.536949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.536913 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:01.643654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.643578 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-wb9cp"] Nov 01 20:07:01.645726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.645705 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:01.671832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.671793 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da45f1cf-786f-43eb-b5f5-0f7f797b3c57-bound-sa-token\") pod \"cert-manager-597b95cb84-wb9cp\" (UID: \"da45f1cf-786f-43eb-b5f5-0f7f797b3c57\") " pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:01.671993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.671838 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhh7\" (UniqueName: \"kubernetes.io/projected/da45f1cf-786f-43eb-b5f5-0f7f797b3c57-kube-api-access-7fhh7\") pod \"cert-manager-597b95cb84-wb9cp\" (UID: \"da45f1cf-786f-43eb-b5f5-0f7f797b3c57\") " pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:01.747005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.746966 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bqhkg\"" Nov 01 20:07:01.748373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.748352 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wb9cp"] Nov 01 20:07:01.772554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.772513 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da45f1cf-786f-43eb-b5f5-0f7f797b3c57-bound-sa-token\") pod \"cert-manager-597b95cb84-wb9cp\" (UID: \"da45f1cf-786f-43eb-b5f5-0f7f797b3c57\") " pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:01.772554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.772555 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhh7\" (UniqueName: \"kubernetes.io/projected/da45f1cf-786f-43eb-b5f5-0f7f797b3c57-kube-api-access-7fhh7\") pod \"cert-manager-597b95cb84-wb9cp\" (UID: \"da45f1cf-786f-43eb-b5f5-0f7f797b3c57\") " pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:01.829089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.829056 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da45f1cf-786f-43eb-b5f5-0f7f797b3c57-bound-sa-token\") pod \"cert-manager-597b95cb84-wb9cp\" (UID: \"da45f1cf-786f-43eb-b5f5-0f7f797b3c57\") " pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:01.829280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.829257 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhh7\" (UniqueName: \"kubernetes.io/projected/da45f1cf-786f-43eb-b5f5-0f7f797b3c57-kube-api-access-7fhh7\") pod \"cert-manager-597b95cb84-wb9cp\" (UID: \"da45f1cf-786f-43eb-b5f5-0f7f797b3c57\") " pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:01.956241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:01.956144 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wb9cp" Nov 01 20:07:02.325153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:02.325128 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wb9cp"] Nov 01 20:07:02.327645 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:07:02.327613 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda45f1cf_786f_43eb_b5f5_0f7f797b3c57.slice/crio-a1cd5d0b08efeb2ff7f0429267c1d896775c92f1e8b5504e248f4800399b76df WatchSource:0}: Error finding container a1cd5d0b08efeb2ff7f0429267c1d896775c92f1e8b5504e248f4800399b76df: Status 404 returned error can't find the container with id a1cd5d0b08efeb2ff7f0429267c1d896775c92f1e8b5504e248f4800399b76df Nov 01 20:07:02.588942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:02.588852 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wb9cp" event={"ID":"da45f1cf-786f-43eb-b5f5-0f7f797b3c57","Type":"ContainerStarted","Data":"0b39fe426fe2d95d3efe0ec5008da71ce4515fff4aaf1133ed382b8790efb129"} Nov 01 20:07:02.588942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:02.588901 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wb9cp" event={"ID":"da45f1cf-786f-43eb-b5f5-0f7f797b3c57","Type":"ContainerStarted","Data":"a1cd5d0b08efeb2ff7f0429267c1d896775c92f1e8b5504e248f4800399b76df"} Nov 01 20:07:02.702882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:02.702828 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-wb9cp" podStartSLOduration=1.7028085960000001 podStartE2EDuration="1.702808596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:02.701381088 +0000 UTC m=+230.375640341" watchObservedRunningTime="2026-11-01 20:07:02.702808596 +0000 UTC m=+230.377067851" Nov 01 20:07:03.492306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:03.492264 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:03.675023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:03.674984 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:04.576930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:04.576894 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nqzfz" Nov 01 20:07:06.474457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:06.474425 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:06.625590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:06.625553 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:06.759898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:06.759801 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:07.173955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:07.173852 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:07.333017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:07.332975 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:07.477717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:07.477684 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:08.589889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.589856 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:08.670419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.670389 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp"] Nov 01 20:07:08.676287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.676263 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.744766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.744724 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 20:07:08.745237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.745205 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l95\" (UniqueName: \"kubernetes.io/projected/1c67a118-8b19-4fe3-8a24-71875f068830-kube-api-access-65l95\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.745387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.745370 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c67a118-8b19-4fe3-8a24-71875f068830-tls-certs\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.745504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.745485 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c67a118-8b19-4fe3-8a24-71875f068830-tmp\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.746379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.746354 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp"] Nov 01 20:07:08.756576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.756545 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:08.779355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.779303 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 20:07:08.779524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.779402 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 20:07:08.779524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.779455 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-db7jz\"" Nov 01 20:07:08.779635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.779561 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 20:07:08.846284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.846186 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c67a118-8b19-4fe3-8a24-71875f068830-tls-certs\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.846284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.846253 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c67a118-8b19-4fe3-8a24-71875f068830-tmp\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.846526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.846300 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l95\" (UniqueName: \"kubernetes.io/projected/1c67a118-8b19-4fe3-8a24-71875f068830-kube-api-access-65l95\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.849052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.849024 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c67a118-8b19-4fe3-8a24-71875f068830-tmp\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.849186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.849121 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c67a118-8b19-4fe3-8a24-71875f068830-tls-certs\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.895795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.895753 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:08.902457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.902431 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l95\" (UniqueName: \"kubernetes.io/projected/1c67a118-8b19-4fe3-8a24-71875f068830-kube-api-access-65l95\") pod \"kube-auth-proxy-5c84dcdb75-9t9pp\" (UID: \"1c67a118-8b19-4fe3-8a24-71875f068830\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:08.989607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:08.989569 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" Nov 01 20:07:09.259174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:09.259139 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:09.361880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:09.361849 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp"] Nov 01 20:07:09.365565 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:07:09.365526 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c67a118_8b19_4fe3_8a24_71875f068830.slice/crio-fd3a65082344d84d065feb22a19005cc39241d813ab0de19ceade49b02a65d6b WatchSource:0}: Error finding container fd3a65082344d84d065feb22a19005cc39241d813ab0de19ceade49b02a65d6b: Status 404 returned error can't find the container with id fd3a65082344d84d065feb22a19005cc39241d813ab0de19ceade49b02a65d6b Nov 01 20:07:09.547525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:09.547437 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:09.622398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:09.622361 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" event={"ID":"1c67a118-8b19-4fe3-8a24-71875f068830","Type":"ContainerStarted","Data":"fd3a65082344d84d065feb22a19005cc39241d813ab0de19ceade49b02a65d6b"} Nov 01 20:07:11.003484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:11.003442 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:11.366063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:11.365975 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:11.490543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:11.490487 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:12.639275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:12.639191 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" event={"ID":"1c67a118-8b19-4fe3-8a24-71875f068830","Type":"ContainerStarted","Data":"88e27b8246ebdfec70806069b7ae65512266ea1742ab6f7050bf5d9283dcfb86"} Nov 01 20:07:12.755822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:12.755770 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-9t9pp" podStartSLOduration=1.780881224 podStartE2EDuration="4.755756606s" totalImagesPullingTime="2.974875382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:12.755128858 +0000 UTC m=+240.429388113" watchObservedRunningTime="2026-11-01 20:07:12.755756606 +0000 UTC m=+240.430015904" Nov 01 20:07:12.854239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:12.854201 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kdxz"] Nov 01 20:07:12.859089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:12.859065 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" Nov 01 20:07:12.912109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:12.911991 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dt6v\"" Nov 01 20:07:12.913507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:12.913475 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kdxz"] Nov 01 20:07:12.986765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:12.986721 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zfn\" (UniqueName: \"kubernetes.io/projected/16a5be80-cf63-4753-b59c-659bbbfa49a9-kube-api-access-t9zfn\") pod \"authorino-57f7f8659f-2kdxz\" (UID: \"16a5be80-cf63-4753-b59c-659bbbfa49a9\") " pod="kuadrant-system/authorino-57f7f8659f-2kdxz" Nov 01 20:07:13.000293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.000246 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:13.088230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.088190 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zfn\" (UniqueName: \"kubernetes.io/projected/16a5be80-cf63-4753-b59c-659bbbfa49a9-kube-api-access-t9zfn\") pod \"authorino-57f7f8659f-2kdxz\" (UID: \"16a5be80-cf63-4753-b59c-659bbbfa49a9\") " pod="kuadrant-system/authorino-57f7f8659f-2kdxz" Nov 01 20:07:13.137521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.137486 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:13.147492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.147467 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zfn\" (UniqueName: \"kubernetes.io/projected/16a5be80-cf63-4753-b59c-659bbbfa49a9-kube-api-access-t9zfn\") pod \"authorino-57f7f8659f-2kdxz\" (UID: \"16a5be80-cf63-4753-b59c-659bbbfa49a9\") " pod="kuadrant-system/authorino-57f7f8659f-2kdxz" Nov 01 20:07:13.171138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.171069 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" Nov 01 20:07:13.539223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.539166 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:13.539709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.539678 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kdxz"] Nov 01 20:07:13.542067 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:07:13.542038 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16a5be80_cf63_4753_b59c_659bbbfa49a9.slice/crio-626328453a8f2aeecc9fb4f333fd28c8f6650063eec3375a9368f4df8300466f WatchSource:0}: Error finding container 626328453a8f2aeecc9fb4f333fd28c8f6650063eec3375a9368f4df8300466f: Status 404 returned error can't find the container with id 626328453a8f2aeecc9fb4f333fd28c8f6650063eec3375a9368f4df8300466f Nov 01 20:07:13.645408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.645371 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" event={"ID":"16a5be80-cf63-4753-b59c-659bbbfa49a9","Type":"ContainerStarted","Data":"626328453a8f2aeecc9fb4f333fd28c8f6650063eec3375a9368f4df8300466f"} Nov 01 20:07:13.668872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:13.668830 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:15.671958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.671923 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:07:15.693791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.693758 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:15.745588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.745553 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z5b7z\"" Nov 01 20:07:15.795275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.795237 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 20:07:15.797144 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.797118 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:07:15.813809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.813774 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnxs\" (UniqueName: \"kubernetes.io/projected/f956c89a-217e-4237-a2cb-70632240750d-kube-api-access-wvnxs\") pod \"limitador-limitador-7c9b87d574-ld4qw\" (UID: \"f956c89a-217e-4237-a2cb-70632240750d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:15.813969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.813819 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f956c89a-217e-4237-a2cb-70632240750d-config-file\") pod \"limitador-limitador-7c9b87d574-ld4qw\" (UID: \"f956c89a-217e-4237-a2cb-70632240750d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:15.915243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.915207 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnxs\" (UniqueName: \"kubernetes.io/projected/f956c89a-217e-4237-a2cb-70632240750d-kube-api-access-wvnxs\") pod \"limitador-limitador-7c9b87d574-ld4qw\" (UID: \"f956c89a-217e-4237-a2cb-70632240750d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:15.915450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.915267 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f956c89a-217e-4237-a2cb-70632240750d-config-file\") pod \"limitador-limitador-7c9b87d574-ld4qw\" (UID: \"f956c89a-217e-4237-a2cb-70632240750d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:15.916066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.916044 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f956c89a-217e-4237-a2cb-70632240750d-config-file\") pod \"limitador-limitador-7c9b87d574-ld4qw\" (UID: \"f956c89a-217e-4237-a2cb-70632240750d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:15.968958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.968921 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:07:15.972791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:15.972762 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnxs\" (UniqueName: \"kubernetes.io/projected/f956c89a-217e-4237-a2cb-70632240750d-kube-api-access-wvnxs\") pod \"limitador-limitador-7c9b87d574-ld4qw\" (UID: \"f956c89a-217e-4237-a2cb-70632240750d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:16.020775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:16.020738 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:16.156720 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:07:16.156682 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:07:16.398719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:16.398530 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:07:16.401845 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:07:16.401813 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf956c89a_217e_4237_a2cb_70632240750d.slice/crio-4ef68ab3aa983070c5e08ec54286121583b7e634ad5ba50e5fc3ecb16e81076c WatchSource:0}: Error finding container 4ef68ab3aa983070c5e08ec54286121583b7e634ad5ba50e5fc3ecb16e81076c: Status 404 returned error can't find the container with id 4ef68ab3aa983070c5e08ec54286121583b7e634ad5ba50e5fc3ecb16e81076c Nov 01 20:07:16.425724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:16.425685 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:16.668273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:16.668184 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" event={"ID":"f956c89a-217e-4237-a2cb-70632240750d","Type":"ContainerStarted","Data":"4ef68ab3aa983070c5e08ec54286121583b7e634ad5ba50e5fc3ecb16e81076c"} Nov 01 20:07:17.674227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:17.674187 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" event={"ID":"16a5be80-cf63-4753-b59c-659bbbfa49a9","Type":"ContainerStarted","Data":"ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5"} Nov 01 20:07:17.786940 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:17.786883 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" podStartSLOduration=2.578285694 podStartE2EDuration="5.786868299s" totalImagesPullingTime="3.208582605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:17.785494852 +0000 UTC m=+245.459754110" watchObservedRunningTime="2026-11-01 20:07:17.786868299 +0000 UTC m=+245.461127599" Nov 01 20:07:18.549307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:18.549270 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:18.681108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:18.681068 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" event={"ID":"f956c89a-217e-4237-a2cb-70632240750d","Type":"ContainerStarted","Data":"db94df79e7bb3445ab6f4e7908942336b5e7e1978ff1d863aede99038485a719"} Nov 01 20:07:18.681581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:18.681276 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:18.846718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:18.846613 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" podStartSLOduration=1.736508903 podStartE2EDuration="3.846597631s" totalImagesPullingTime="2.110088728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:18.845165482 +0000 UTC m=+246.519424738" watchObservedRunningTime="2026-11-01 20:07:18.846597631 +0000 UTC m=+246.520856886" Nov 01 20:07:20.143728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.143691 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kdxz"] Nov 01 20:07:20.144123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.143905 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" podUID="16a5be80-cf63-4753-b59c-659bbbfa49a9" containerName="authorino" containerID="cri-o://ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5" gracePeriod=30 Nov 01 20:07:20.389009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.388979 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" Nov 01 20:07:20.453969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.453878 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a5be80-cf63-4753-b59c-659bbbfa49a9-kube-api-access-t9zfn\" (UniqueName: \"kubernetes.io/projected/16a5be80-cf63-4753-b59c-659bbbfa49a9-kube-api-access-t9zfn\") pod \"16a5be80-cf63-4753-b59c-659bbbfa49a9\" (UID: \"16a5be80-cf63-4753-b59c-659bbbfa49a9\") " Nov 01 20:07:20.456181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.456147 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a5be80-cf63-4753-b59c-659bbbfa49a9-kube-api-access-t9zfn" pod "16a5be80-cf63-4753-b59c-659bbbfa49a9" (UID: "16a5be80-cf63-4753-b59c-659bbbfa49a9"). InnerVolumeSpecName "kube-api-access-t9zfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:07:20.555025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.554906 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zfn\" (UniqueName: \"kubernetes.io/projected/16a5be80-cf63-4753-b59c-659bbbfa49a9-kube-api-access-t9zfn\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:07:20.692672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.692631 5633 generic.go:356] "Generic (PLEG): container finished" podID="16a5be80-cf63-4753-b59c-659bbbfa49a9" containerID="ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5" exitCode=0 Nov 01 20:07:20.692672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.692677 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" Nov 01 20:07:20.692879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.692711 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" event={"ID":"16a5be80-cf63-4753-b59c-659bbbfa49a9","Type":"ContainerDied","Data":"ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5"} Nov 01 20:07:20.692879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.692753 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2kdxz" event={"ID":"16a5be80-cf63-4753-b59c-659bbbfa49a9","Type":"ContainerDied","Data":"626328453a8f2aeecc9fb4f333fd28c8f6650063eec3375a9368f4df8300466f"} Nov 01 20:07:20.692879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.692772 5633 scope.go:122] "RemoveContainer" containerID="ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5" Nov 01 20:07:20.701972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.701949 5633 scope.go:122] "RemoveContainer" containerID="ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5" Nov 01 20:07:20.702345 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:07:20.702304 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5\": container with ID starting with ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5 not found: ID does not exist" containerID="ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5" Nov 01 20:07:20.702399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.702368 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5"} err="failed to get container status \"ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5\": rpc error: code = NotFound desc = could not find container \"ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5\": container with ID starting with ef6416d8f1a412f8cca9e9ebd0406affd5522b2efb846818fc8397d8fed625f5 not found: ID does not exist" Nov 01 20:07:20.854017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.853978 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kdxz"] Nov 01 20:07:20.857294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:20.857259 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2kdxz"] Nov 01 20:07:22.464764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:22.464726 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a5be80-cf63-4753-b59c-659bbbfa49a9" path="/var/lib/kubelet/pods/16a5be80-cf63-4753-b59c-659bbbfa49a9/volumes" Nov 01 20:07:29.687052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:29.687012 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ld4qw" Nov 01 20:07:29.962524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:29.962489 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:30.101346 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:30.101305 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:30.237135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:30.237044 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:38.895803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:38.895759 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:39.111988 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:39.111949 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:39.632517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:39.632471 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:40.228463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:40.228428 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:40.405248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:40.405208 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:46.156819 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:07:46.156779 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:07:59.768322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.768229 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:59.834009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.833971 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b77d64cf-snt6h"] Nov 01 20:07:59.834343 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.834316 5633 cpu_manager.go:414] "containerMap: removing container" podUID="16a5be80-cf63-4753-b59c-659bbbfa49a9" containerName="authorino" Nov 01 20:07:59.834391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.834345 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a5be80-cf63-4753-b59c-659bbbfa49a9" containerName="authorino" Nov 01 20:07:59.834440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.834431 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a5be80-cf63-4753-b59c-659bbbfa49a9" containerName="authorino" Nov 01 20:07:59.837670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.837649 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:07:59.890148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.890121 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 20:07:59.894853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.894810 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:07:59.899033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.899012 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4sm\" (UniqueName: \"kubernetes.io/projected/847b8795-c21f-4757-9164-848f0a3d0273-kube-api-access-qf4sm\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:07:59.899177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.899155 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-webhook-cert\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:07:59.899249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.899200 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-metrics-certs\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:07:59.939702 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.939667 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 20:07:59.939702 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.939667 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 20:07:59.939933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.939863 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 20:07:59.939933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.939897 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 20:07:59.939933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.939928 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5ch2z\"" Nov 01 20:07:59.941725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.941701 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b77d64cf-snt6h"] Nov 01 20:07:59.999878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.999842 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-webhook-cert\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:00.000080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.999929 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-metrics-certs\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:00.000080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:07:59.999971 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4sm\" (UniqueName: \"kubernetes.io/projected/847b8795-c21f-4757-9164-848f0a3d0273-kube-api-access-qf4sm\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:00.002697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.002670 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-metrics-certs\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:00.002824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.002726 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-webhook-cert\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:00.021895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.021813 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:00.057472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.057439 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4sm\" (UniqueName: \"kubernetes.io/projected/847b8795-c21f-4757-9164-848f0a3d0273-kube-api-access-qf4sm\") pod \"maas-controller-5b77d64cf-snt6h\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:00.144563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.144523 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:00.150032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.150011 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:00.384944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.384854 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645c58c6f7-mc6zb"] Nov 01 20:08:00.388885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.388858 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.403529 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.403498 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-metrics-certs\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.403692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.403555 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-webhook-cert\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.403692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.403658 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwctv\" (UniqueName: \"kubernetes.io/projected/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-kube-api-access-pwctv\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.448663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.448552 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-mc6zb"] Nov 01 20:08:00.504716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.504678 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-metrics-certs\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.504920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.504737 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-webhook-cert\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.504920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.504762 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwctv\" (UniqueName: \"kubernetes.io/projected/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-kube-api-access-pwctv\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.507569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.507539 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-webhook-cert\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.507748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.507655 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-metrics-certs\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.561994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.561959 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwctv\" (UniqueName: \"kubernetes.io/projected/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-kube-api-access-pwctv\") pod \"maas-controller-645c58c6f7-mc6zb\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.570037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.570010 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b77d64cf-snt6h"] Nov 01 20:08:00.572093 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:08:00.572063 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847b8795_c21f_4757_9164_848f0a3d0273.slice/crio-0cd6b16395f2070158249bab646779ea87ffeb7619ef8c5cb18cd3498f8a4689 WatchSource:0}: Error finding container 0cd6b16395f2070158249bab646779ea87ffeb7619ef8c5cb18cd3498f8a4689: Status 404 returned error can't find the container with id 0cd6b16395f2070158249bab646779ea87ffeb7619ef8c5cb18cd3498f8a4689 Nov 01 20:08:00.706106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.706007 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:00.890363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:00.890318 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b77d64cf-snt6h" event={"ID":"847b8795-c21f-4757-9164-848f0a3d0273","Type":"ContainerStarted","Data":"0cd6b16395f2070158249bab646779ea87ffeb7619ef8c5cb18cd3498f8a4689"} Nov 01 20:08:01.082876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:01.082851 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-mc6zb"] Nov 01 20:08:01.084685 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:08:01.084659 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ae1fa9_afa7_4162_b9e7_28bf4d6c2864.slice/crio-ac957764179a61eaa3dcb395d0f241e822ab6e264acb799fcbdb70f0f181566f WatchSource:0}: Error finding container ac957764179a61eaa3dcb395d0f241e822ab6e264acb799fcbdb70f0f181566f: Status 404 returned error can't find the container with id ac957764179a61eaa3dcb395d0f241e822ab6e264acb799fcbdb70f0f181566f Nov 01 20:08:01.896793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:01.896757 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" event={"ID":"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864","Type":"ContainerStarted","Data":"ac957764179a61eaa3dcb395d0f241e822ab6e264acb799fcbdb70f0f181566f"} Nov 01 20:08:07.491687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:07.491654 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:07.633197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:07.633156 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:07.929848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:07.929764 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b77d64cf-snt6h" event={"ID":"847b8795-c21f-4757-9164-848f0a3d0273","Type":"ContainerStarted","Data":"72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2"} Nov 01 20:08:07.929848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:07.929832 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:07.931269 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:07.931247 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" event={"ID":"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864","Type":"ContainerStarted","Data":"4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e"} Nov 01 20:08:07.931390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:07.931380 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:08.062058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:08.062001 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5b77d64cf-snt6h" podStartSLOduration=2.145519166 podStartE2EDuration="9.061984757s" totalImagesPullingTime="6.916465591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:08.059814704 +0000 UTC m=+295.734073958" watchObservedRunningTime="2026-11-01 20:08:08.061984757 +0000 UTC m=+295.736244012" Nov 01 20:08:08.197112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:08.197008 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podStartSLOduration=2.195866684 podStartE2EDuration="8.196993178s" totalImagesPullingTime="6.001126494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:08.195402343 +0000 UTC m=+295.869661598" watchObservedRunningTime="2026-11-01 20:08:08.196993178 +0000 UTC m=+295.871252470" Nov 01 20:08:10.546650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.546613 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47"] Nov 01 20:08:10.550294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.550269 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.596486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.596440 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtzj\" (UniqueName: \"kubernetes.io/projected/292d7509-aef9-452f-ab7f-85e1117d9652-kube-api-access-svtzj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.596688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.596500 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.596688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.596578 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.596788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.596687 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/292d7509-aef9-452f-ab7f-85e1117d9652-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.606664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.606621 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47"] Nov 01 20:08:10.654523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.654476 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 20:08:10.654523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.654498 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 20:08:10.654523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.654511 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 20:08:10.654824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.654580 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-knqbm\"" Nov 01 20:08:10.698139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.698105 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/292d7509-aef9-452f-ab7f-85e1117d9652-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.698374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.698205 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtzj\" (UniqueName: \"kubernetes.io/projected/292d7509-aef9-452f-ab7f-85e1117d9652-kube-api-access-svtzj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.698374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.698245 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.698374 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:10.698265 5633 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 01 20:08:10.698374 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:10.698325 5633 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 01 20:08:10.698374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.698272 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.698616 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:10.698390 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/292d7509-aef9-452f-ab7f-85e1117d9652-tls-cert podName:292d7509-aef9-452f-ab7f-85e1117d9652 nodeName:}" failed. No retries permitted until 2026-11-01 20:08:11.198364045 +0000 UTC m=+298.872623301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/292d7509-aef9-452f-ab7f-85e1117d9652-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-vjv47" (UID: "292d7509-aef9-452f-ab7f-85e1117d9652") : secret "usage-logs-tenancy-proxy-tls" not found Nov 01 20:08:10.698616 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:10.698427 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-proxy-source podName:292d7509-aef9-452f-ab7f-85e1117d9652 nodeName:}" failed. No retries permitted until 2026-11-01 20:08:11.198408428 +0000 UTC m=+298.872667694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-vjv47" (UID: "292d7509-aef9-452f-ab7f-85e1117d9652") : configmap "usage-tenancy-proxy-source" not found Nov 01 20:08:10.698889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.698869 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.700564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.700522 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:10.756201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.756163 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtzj\" (UniqueName: \"kubernetes.io/projected/292d7509-aef9-452f-ab7f-85e1117d9652-kube-api-access-svtzj\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:10.948239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:10.948155 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:11.202965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:11.202878 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/292d7509-aef9-452f-ab7f-85e1117d9652-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:11.203122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:11.203003 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:11.203827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:11.203802 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/292d7509-aef9-452f-ab7f-85e1117d9652-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:11.205651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:11.205626 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/292d7509-aef9-452f-ab7f-85e1117d9652-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjv47\" (UID: \"292d7509-aef9-452f-ab7f-85e1117d9652\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:11.463225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:11.463193 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:11.853423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:11.853393 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47"] Nov 01 20:08:11.855307 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:08:11.855278 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292d7509_aef9_452f_ab7f_85e1117d9652.slice/crio-986ead255903d014686567321379a26fbd821b43afca83fb178f9798470a9941 WatchSource:0}: Error finding container 986ead255903d014686567321379a26fbd821b43afca83fb178f9798470a9941: Status 404 returned error can't find the container with id 986ead255903d014686567321379a26fbd821b43afca83fb178f9798470a9941 Nov 01 20:08:11.951923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:11.951885 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" event={"ID":"292d7509-aef9-452f-ab7f-85e1117d9652","Type":"ContainerStarted","Data":"986ead255903d014686567321379a26fbd821b43afca83fb178f9798470a9941"} Nov 01 20:08:16.167943 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:16.167880 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:08:18.940352 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:18.940299 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:08:18.940836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:18.940550 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:19.047389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.047354 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:19.110227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.110188 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b77d64cf-snt6h"] Nov 01 20:08:19.110548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.110521 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5b77d64cf-snt6h" podUID="847b8795-c21f-4757-9164-848f0a3d0273" containerName="manager" containerID="cri-o://72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2" gracePeriod=30 Nov 01 20:08:19.386498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.386468 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:19.496996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.496905 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-webhook-cert\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-webhook-cert\") pod \"847b8795-c21f-4757-9164-848f0a3d0273\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " Nov 01 20:08:19.496996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.496981 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847b8795-c21f-4757-9164-848f0a3d0273-kube-api-access-qf4sm\" (UniqueName: \"kubernetes.io/projected/847b8795-c21f-4757-9164-848f0a3d0273-kube-api-access-qf4sm\") pod \"847b8795-c21f-4757-9164-848f0a3d0273\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " Nov 01 20:08:19.497228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.497023 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-metrics-certs\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-metrics-certs\") pod \"847b8795-c21f-4757-9164-848f0a3d0273\" (UID: \"847b8795-c21f-4757-9164-848f0a3d0273\") " Nov 01 20:08:19.502373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.500007 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-metrics-certs" pod "847b8795-c21f-4757-9164-848f0a3d0273" (UID: "847b8795-c21f-4757-9164-848f0a3d0273"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:08:19.502373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.500222 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847b8795-c21f-4757-9164-848f0a3d0273-kube-api-access-qf4sm" pod "847b8795-c21f-4757-9164-848f0a3d0273" (UID: "847b8795-c21f-4757-9164-848f0a3d0273"). InnerVolumeSpecName "kube-api-access-qf4sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:08:19.525586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.519467 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-webhook-cert" pod "847b8795-c21f-4757-9164-848f0a3d0273" (UID: "847b8795-c21f-4757-9164-848f0a3d0273"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:08:19.598316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.598278 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4sm\" (UniqueName: \"kubernetes.io/projected/847b8795-c21f-4757-9164-848f0a3d0273-kube-api-access-qf4sm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:19.598316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.598315 5633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-metrics-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:19.598564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.598326 5633 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/847b8795-c21f-4757-9164-848f0a3d0273-webhook-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:19.998436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.998398 5633 generic.go:356] "Generic (PLEG): container finished" podID="847b8795-c21f-4757-9164-848f0a3d0273" containerID="72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2" exitCode=0 Nov 01 20:08:19.998924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.998452 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b77d64cf-snt6h" event={"ID":"847b8795-c21f-4757-9164-848f0a3d0273","Type":"ContainerDied","Data":"72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2"} Nov 01 20:08:19.998924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.998459 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b77d64cf-snt6h" Nov 01 20:08:19.998924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.998502 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b77d64cf-snt6h" event={"ID":"847b8795-c21f-4757-9164-848f0a3d0273","Type":"ContainerDied","Data":"0cd6b16395f2070158249bab646779ea87ffeb7619ef8c5cb18cd3498f8a4689"} Nov 01 20:08:19.998924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:19.998531 5633 scope.go:122] "RemoveContainer" containerID="72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2" Nov 01 20:08:20.010254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:20.010229 5633 scope.go:122] "RemoveContainer" containerID="72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2" Nov 01 20:08:20.010654 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:20.010628 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2\": container with ID starting with 72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2 not found: ID does not exist" containerID="72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2" Nov 01 20:08:20.010737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:20.010671 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2"} err="failed to get container status \"72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2\": rpc error: code = NotFound desc = could not find container \"72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2\": container with ID starting with 72f4df5bc997819099918cd38e4dd1c87dc085a2285185554c845998b87a30c2 not found: ID does not exist" Nov 01 20:08:20.176446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:20.176409 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b77d64cf-snt6h"] Nov 01 20:08:20.227216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:20.227180 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b77d64cf-snt6h"] Nov 01 20:08:20.469819 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:20.469775 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847b8795-c21f-4757-9164-848f0a3d0273" path="/var/lib/kubelet/pods/847b8795-c21f-4757-9164-848f0a3d0273/volumes" Nov 01 20:08:26.545504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:26.545453 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:26.733588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:26.733549 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:26.958708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:26.958598 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:27.091041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.091010 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:27.216108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.216068 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:27.350067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.350023 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:27.447506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.447472 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4dzk7"] Nov 01 20:08:27.447857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.447844 5633 cpu_manager.go:414] "containerMap: removing container" podUID="847b8795-c21f-4757-9164-848f0a3d0273" containerName="manager" Nov 01 20:08:27.447857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.447857 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847b8795-c21f-4757-9164-848f0a3d0273" containerName="manager" Nov 01 20:08:27.447937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.447919 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="847b8795-c21f-4757-9164-848f0a3d0273" containerName="manager" Nov 01 20:08:27.451088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.451069 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.484213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.484133 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:27.507701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.507672 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4dzk7"] Nov 01 20:08:27.554900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.554865 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 20:08:27.555317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.555012 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7tn6l\"" Nov 01 20:08:27.580600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.580567 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae182c81-580e-4e54-bf4b-9696750c18ed-cert\") pod \"kserve-controller-manager-776cc66d47-4dzk7\" (UID: \"ae182c81-580e-4e54-bf4b-9696750c18ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.580770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.580633 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2wc\" (UniqueName: \"kubernetes.io/projected/ae182c81-580e-4e54-bf4b-9696750c18ed-kube-api-access-ld2wc\") pod \"kserve-controller-manager-776cc66d47-4dzk7\" (UID: \"ae182c81-580e-4e54-bf4b-9696750c18ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.611268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.611234 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:27.681974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.681939 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae182c81-580e-4e54-bf4b-9696750c18ed-cert\") pod \"kserve-controller-manager-776cc66d47-4dzk7\" (UID: \"ae182c81-580e-4e54-bf4b-9696750c18ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.682183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.682041 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2wc\" (UniqueName: \"kubernetes.io/projected/ae182c81-580e-4e54-bf4b-9696750c18ed-kube-api-access-ld2wc\") pod \"kserve-controller-manager-776cc66d47-4dzk7\" (UID: \"ae182c81-580e-4e54-bf4b-9696750c18ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.684633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.684606 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nscg4"] Nov 01 20:08:27.684868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.684849 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae182c81-580e-4e54-bf4b-9696750c18ed-cert\") pod \"kserve-controller-manager-776cc66d47-4dzk7\" (UID: \"ae182c81-580e-4e54-bf4b-9696750c18ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.700852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.700825 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:27.738689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.738619 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2wc\" (UniqueName: \"kubernetes.io/projected/ae182c81-580e-4e54-bf4b-9696750c18ed-kube-api-access-ld2wc\") pod \"kserve-controller-manager-776cc66d47-4dzk7\" (UID: \"ae182c81-580e-4e54-bf4b-9696750c18ed\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.744932 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.744898 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nscg4"] Nov 01 20:08:27.752564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.752536 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 20:08:27.752564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.752541 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ck4qq\"" Nov 01 20:08:27.763898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.763864 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:27.783514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.783479 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775c372c-52f9-4571-b7c1-28905e6f854c-tls-certs\") pod \"model-serving-api-75bd89fdfc-nscg4\" (UID: \"775c372c-52f9-4571-b7c1-28905e6f854c\") " pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:27.783655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.783547 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77hm\" (UniqueName: \"kubernetes.io/projected/775c372c-52f9-4571-b7c1-28905e6f854c-kube-api-access-q77hm\") pod \"model-serving-api-75bd89fdfc-nscg4\" (UID: \"775c372c-52f9-4571-b7c1-28905e6f854c\") " pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:27.792085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.792048 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-wpr7v"] Nov 01 20:08:27.795846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.795819 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.851646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.851613 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wpr7v"] Nov 01 20:08:27.852755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.852726 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 20:08:27.852897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.852766 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 20:08:27.852897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.852797 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-f5888\"" Nov 01 20:08:27.884823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.884791 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775c372c-52f9-4571-b7c1-28905e6f854c-tls-certs\") pod \"model-serving-api-75bd89fdfc-nscg4\" (UID: \"775c372c-52f9-4571-b7c1-28905e6f854c\") " pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:27.885029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.884838 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77hm\" (UniqueName: \"kubernetes.io/projected/775c372c-52f9-4571-b7c1-28905e6f854c-kube-api-access-q77hm\") pod \"model-serving-api-75bd89fdfc-nscg4\" (UID: \"775c372c-52f9-4571-b7c1-28905e6f854c\") " pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:27.885029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.884871 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5778aa7-6472-4e46-afa1-4048cb03ef96-metrics-certs\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.885029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.884918 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5778aa7-6472-4e46-afa1-4048cb03ef96-cert\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.885029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.884949 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twq5\" (UniqueName: \"kubernetes.io/projected/d5778aa7-6472-4e46-afa1-4048cb03ef96-kube-api-access-9twq5\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.887542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.887516 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775c372c-52f9-4571-b7c1-28905e6f854c-tls-certs\") pod \"model-serving-api-75bd89fdfc-nscg4\" (UID: \"775c372c-52f9-4571-b7c1-28905e6f854c\") " pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:27.941888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.941850 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77hm\" (UniqueName: \"kubernetes.io/projected/775c372c-52f9-4571-b7c1-28905e6f854c-kube-api-access-q77hm\") pod \"model-serving-api-75bd89fdfc-nscg4\" (UID: \"775c372c-52f9-4571-b7c1-28905e6f854c\") " pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:27.985664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.985629 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5778aa7-6472-4e46-afa1-4048cb03ef96-cert\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.985872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.985688 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twq5\" (UniqueName: \"kubernetes.io/projected/d5778aa7-6472-4e46-afa1-4048cb03ef96-kube-api-access-9twq5\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.985872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.985767 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5778aa7-6472-4e46-afa1-4048cb03ef96-metrics-certs\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.988391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.988360 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5778aa7-6472-4e46-afa1-4048cb03ef96-cert\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:27.988517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:27.988362 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5778aa7-6472-4e46-afa1-4048cb03ef96-metrics-certs\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:28.014378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.014286 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:28.042040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.042006 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twq5\" (UniqueName: \"kubernetes.io/projected/d5778aa7-6472-4e46-afa1-4048cb03ef96-kube-api-access-9twq5\") pod \"odh-model-controller-769875db56-wpr7v\" (UID: \"d5778aa7-6472-4e46-afa1-4048cb03ef96\") " pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:28.048134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.048100 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" event={"ID":"292d7509-aef9-452f-ab7f-85e1117d9652","Type":"ContainerStarted","Data":"c24fb00ece9024a19f0553141d4cc4583efd979b5a4c7735cc7007b51b952d96"} Nov 01 20:08:28.048311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.048182 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:28.110209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.110176 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:28.138535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.138499 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4dzk7"] Nov 01 20:08:28.142763 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:08:28.142730 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae182c81_580e_4e54_bf4b_9696750c18ed.slice/crio-0d55536800229e5869b0b4fb246e9fef047d736562da932c1be8b58d90a56a71 WatchSource:0}: Error finding container 0d55536800229e5869b0b4fb246e9fef047d736562da932c1be8b58d90a56a71: Status 404 returned error can't find the container with id 0d55536800229e5869b0b4fb246e9fef047d736562da932c1be8b58d90a56a71 Nov 01 20:08:28.144354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.144302 5633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:08:28.162359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.162240 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" podStartSLOduration=2.793479718 podStartE2EDuration="18.162218488s" totalImagesPullingTime="15.36873877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:28.160761238 +0000 UTC m=+315.835020504" watchObservedRunningTime="2026-11-01 20:08:28.162218488 +0000 UTC m=+315.836477736" Nov 01 20:08:28.395137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.395109 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nscg4"] Nov 01 20:08:28.397225 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:08:28.397191 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775c372c_52f9_4571_b7c1_28905e6f854c.slice/crio-f51462ea1b3cced3ff0d55c07564e47b18d8b5f41573bd15067bbeadbe07a77f WatchSource:0}: Error finding container f51462ea1b3cced3ff0d55c07564e47b18d8b5f41573bd15067bbeadbe07a77f: Status 404 returned error can't find the container with id f51462ea1b3cced3ff0d55c07564e47b18d8b5f41573bd15067bbeadbe07a77f Nov 01 20:08:28.487585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.487497 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wpr7v"] Nov 01 20:08:28.489736 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:08:28.489702 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5778aa7_6472_4e46_afa1_4048cb03ef96.slice/crio-6ee068e9d4a7d00a926ba94012fc49cc5fbab8f5e5f67630853d2d2609d5a6c5 WatchSource:0}: Error finding container 6ee068e9d4a7d00a926ba94012fc49cc5fbab8f5e5f67630853d2d2609d5a6c5: Status 404 returned error can't find the container with id 6ee068e9d4a7d00a926ba94012fc49cc5fbab8f5e5f67630853d2d2609d5a6c5 Nov 01 20:08:28.809482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:28.809444 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:29.055580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:29.055538 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" event={"ID":"775c372c-52f9-4571-b7c1-28905e6f854c","Type":"ContainerStarted","Data":"f51462ea1b3cced3ff0d55c07564e47b18d8b5f41573bd15067bbeadbe07a77f"} Nov 01 20:08:29.057166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:29.057129 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wpr7v" event={"ID":"d5778aa7-6472-4e46-afa1-4048cb03ef96","Type":"ContainerStarted","Data":"6ee068e9d4a7d00a926ba94012fc49cc5fbab8f5e5f67630853d2d2609d5a6c5"} Nov 01 20:08:29.058600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:29.058571 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" event={"ID":"ae182c81-580e-4e54-bf4b-9696750c18ed","Type":"ContainerStarted","Data":"0d55536800229e5869b0b4fb246e9fef047d736562da932c1be8b58d90a56a71"} Nov 01 20:08:32.079580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.079540 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" event={"ID":"775c372c-52f9-4571-b7c1-28905e6f854c","Type":"ContainerStarted","Data":"cd1c9a653f32ac56dca3dbffb9889f2726b669595325252af68d02454546b025"} Nov 01 20:08:32.080007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.079702 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:32.081406 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.081375 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wpr7v" event={"ID":"d5778aa7-6472-4e46-afa1-4048cb03ef96","Type":"ContainerStarted","Data":"b07d745da080244fbc62962343c3d1daf889d3e4c77588433908f4666b4d9586"} Nov 01 20:08:32.081547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.081431 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:32.083106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.083073 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" event={"ID":"ae182c81-580e-4e54-bf4b-9696750c18ed","Type":"ContainerStarted","Data":"1bb86b7026a6ac6ede0f2f938c1a62663f43d5759e8430d3abe1e0270654219d"} Nov 01 20:08:32.083216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.083201 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:08:32.195444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.195377 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" podStartSLOduration=2.3983654469999998 podStartE2EDuration="5.195356669s" totalImagesPullingTime="2.796991222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:32.193614706 +0000 UTC m=+319.867874152" watchObservedRunningTime="2026-11-01 20:08:32.195356669 +0000 UTC m=+319.869615926" Nov 01 20:08:32.314407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.314283 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" podStartSLOduration=2.313604412 podStartE2EDuration="5.314268282s" totalImagesPullingTime="3.00066387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:32.3118424 +0000 UTC m=+319.986101660" watchObservedRunningTime="2026-11-01 20:08:32.314268282 +0000 UTC m=+319.988527567" Nov 01 20:08:32.427206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:32.427149 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-wpr7v" podStartSLOduration=1.9073707739999999 podStartE2EDuration="5.427134189s" totalImagesPullingTime="3.519763415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:32.426055946 +0000 UTC m=+320.100315201" watchObservedRunningTime="2026-11-01 20:08:32.427134189 +0000 UTC m=+320.101393459" Nov 01 20:08:33.987661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:33.987623 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:34.223323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:34.223289 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:34.351408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:34.351288 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:34.471675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:34.471636 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:34.602527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:34.602442 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:34.727443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:34.727400 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:34.856805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:34.856715 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:34.987618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:34.987584 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:35.111910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:35.111816 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:35.236746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:35.236691 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:39.064974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:39.064934 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjv47" Nov 01 20:08:40.793557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:40.793525 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:40.925832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:40.925788 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:42.866691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:42.866652 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:43.038521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:43.038471 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:43.091018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:43.090988 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-wpr7v" Nov 01 20:08:43.092954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:43.092928 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-nscg4" Nov 01 20:08:43.183218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:43.183116 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:43.310277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:43.310229 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:46.156249 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:46.156212 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:08:49.480444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:49.480409 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:08:58.852219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:58.852181 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:59.142758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.142679 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5xl6p"] Nov 01 20:08:59.145162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.145130 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:08:59.364164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.364122 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5xl6p"] Nov 01 20:08:59.364387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.364229 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" Nov 01 20:08:59.437891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.437815 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dt6v\"" Nov 01 20:08:59.481127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.481089 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2sr\" (UniqueName: \"kubernetes.io/projected/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7-kube-api-access-ns2sr\") pod \"authorino-6f6944bcdd-5xl6p\" (UID: \"ac3f0cad-76fd-4ae4-9f2c-5655860b1da7\") " pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" Nov 01 20:08:59.582201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.582164 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2sr\" (UniqueName: \"kubernetes.io/projected/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7-kube-api-access-ns2sr\") pod \"authorino-6f6944bcdd-5xl6p\" (UID: \"ac3f0cad-76fd-4ae4-9f2c-5655860b1da7\") " pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" Nov 01 20:08:59.622008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.621970 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5xl6p"] Nov 01 20:08:59.622233 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:08:59.622213 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ns2sr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" podUID="ac3f0cad-76fd-4ae4-9f2c-5655860b1da7" Nov 01 20:08:59.651608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.651568 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2sr\" (UniqueName: \"kubernetes.io/projected/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7-kube-api-access-ns2sr\") pod \"authorino-6f6944bcdd-5xl6p\" (UID: \"ac3f0cad-76fd-4ae4-9f2c-5655860b1da7\") " pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" Nov 01 20:08:59.707849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.707770 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7464c76956-bvlfb"] Nov 01 20:08:59.738101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.738075 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7464c76956-bvlfb" Nov 01 20:08:59.768272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.768235 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7464c76956-bvlfb"] Nov 01 20:08:59.885722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.885689 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7xr\" (UniqueName: \"kubernetes.io/projected/1a282977-bcc4-4219-ae4d-b79f1e2f2c74-kube-api-access-ts7xr\") pod \"authorino-7464c76956-bvlfb\" (UID: \"1a282977-bcc4-4219-ae4d-b79f1e2f2c74\") " pod="kuadrant-system/authorino-7464c76956-bvlfb" Nov 01 20:08:59.986391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:08:59.986359 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7xr\" (UniqueName: \"kubernetes.io/projected/1a282977-bcc4-4219-ae4d-b79f1e2f2c74-kube-api-access-ts7xr\") pod \"authorino-7464c76956-bvlfb\" (UID: \"1a282977-bcc4-4219-ae4d-b79f1e2f2c74\") " pod="kuadrant-system/authorino-7464c76956-bvlfb" Nov 01 20:09:00.038006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.037968 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7464c76956-bvlfb"] Nov 01 20:09:00.038280 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:09:00.038249 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ts7xr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7464c76956-bvlfb" podUID="1a282977-bcc4-4219-ae4d-b79f1e2f2c74" Nov 01 20:09:00.043525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.043496 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7xr\" (UniqueName: \"kubernetes.io/projected/1a282977-bcc4-4219-ae4d-b79f1e2f2c74-kube-api-access-ts7xr\") pod \"authorino-7464c76956-bvlfb\" (UID: \"1a282977-bcc4-4219-ae4d-b79f1e2f2c74\") " pod="kuadrant-system/authorino-7464c76956-bvlfb" Nov 01 20:09:00.065528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.065493 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bb6ff9d46-d6s2q"] Nov 01 20:09:00.067956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.067937 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.122887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.122853 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 20:09:00.124667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.124639 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bb6ff9d46-d6s2q"] Nov 01 20:09:00.188422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.188385 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-tls-cert\") pod \"authorino-7bb6ff9d46-d6s2q\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.188600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.188520 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ckm\" (UniqueName: \"kubernetes.io/projected/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-kube-api-access-67ckm\") pod \"authorino-7bb6ff9d46-d6s2q\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.224289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.224255 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" Nov 01 20:09:00.224506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.224255 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7464c76956-bvlfb" Nov 01 20:09:00.230636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.230612 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" Nov 01 20:09:00.234147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.234124 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7464c76956-bvlfb" Nov 01 20:09:00.289885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.289804 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7-kube-api-access-ns2sr\" (UniqueName: \"kubernetes.io/projected/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7-kube-api-access-ns2sr\") pod \"ac3f0cad-76fd-4ae4-9f2c-5655860b1da7\" (UID: \"ac3f0cad-76fd-4ae4-9f2c-5655860b1da7\") " Nov 01 20:09:00.289885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.289879 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a282977-bcc4-4219-ae4d-b79f1e2f2c74-kube-api-access-ts7xr\" (UniqueName: \"kubernetes.io/projected/1a282977-bcc4-4219-ae4d-b79f1e2f2c74-kube-api-access-ts7xr\") pod \"1a282977-bcc4-4219-ae4d-b79f1e2f2c74\" (UID: \"1a282977-bcc4-4219-ae4d-b79f1e2f2c74\") " Nov 01 20:09:00.290125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.290010 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ckm\" (UniqueName: \"kubernetes.io/projected/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-kube-api-access-67ckm\") pod \"authorino-7bb6ff9d46-d6s2q\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.290125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.290075 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-tls-cert\") pod \"authorino-7bb6ff9d46-d6s2q\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.292124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.292085 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a282977-bcc4-4219-ae4d-b79f1e2f2c74-kube-api-access-ts7xr" pod "1a282977-bcc4-4219-ae4d-b79f1e2f2c74" (UID: "1a282977-bcc4-4219-ae4d-b79f1e2f2c74"). InnerVolumeSpecName "kube-api-access-ts7xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:09:00.292124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.292111 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7-kube-api-access-ns2sr" pod "ac3f0cad-76fd-4ae4-9f2c-5655860b1da7" (UID: "ac3f0cad-76fd-4ae4-9f2c-5655860b1da7"). InnerVolumeSpecName "kube-api-access-ns2sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:09:00.292576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.292557 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-tls-cert\") pod \"authorino-7bb6ff9d46-d6s2q\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.347130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.347093 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ckm\" (UniqueName: \"kubernetes.io/projected/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-kube-api-access-67ckm\") pod \"authorino-7bb6ff9d46-d6s2q\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.379852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.379813 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:09:00.391000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.390974 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts7xr\" (UniqueName: \"kubernetes.io/projected/1a282977-bcc4-4219-ae4d-b79f1e2f2c74-kube-api-access-ts7xr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:09:00.391063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.391005 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2sr\" (UniqueName: \"kubernetes.io/projected/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7-kube-api-access-ns2sr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:09:00.747438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:00.747381 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bb6ff9d46-d6s2q"] Nov 01 20:09:00.749733 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:09:00.749703 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0aeb571_0ee2_4503_bef9_e22ea7dd373c.slice/crio-fa7c772c80c293bfd615d7b596078cdf997d6a8e1327b82d93d864ab0f194f7e WatchSource:0}: Error finding container fa7c772c80c293bfd615d7b596078cdf997d6a8e1327b82d93d864ab0f194f7e: Status 404 returned error can't find the container with id fa7c772c80c293bfd615d7b596078cdf997d6a8e1327b82d93d864ab0f194f7e Nov 01 20:09:01.229926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:01.229894 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7464c76956-bvlfb" Nov 01 20:09:01.230388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:01.229922 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" event={"ID":"f0aeb571-0ee2-4503-bef9-e22ea7dd373c","Type":"ContainerStarted","Data":"fa7c772c80c293bfd615d7b596078cdf997d6a8e1327b82d93d864ab0f194f7e"} Nov 01 20:09:01.230388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:01.230099 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-5xl6p" Nov 01 20:09:01.451087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:01.451050 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5xl6p"] Nov 01 20:09:01.452740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:01.452711 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-5xl6p"] Nov 01 20:09:01.623205 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:01.623105 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7464c76956-bvlfb"] Nov 01 20:09:01.624952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:01.624928 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7464c76956-bvlfb"] Nov 01 20:09:02.465682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:02.465648 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a282977-bcc4-4219-ae4d-b79f1e2f2c74" path="/var/lib/kubelet/pods/1a282977-bcc4-4219-ae4d-b79f1e2f2c74/volumes" Nov 01 20:09:02.466062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:02.465880 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3f0cad-76fd-4ae4-9f2c-5655860b1da7" path="/var/lib/kubelet/pods/ac3f0cad-76fd-4ae4-9f2c-5655860b1da7/volumes" Nov 01 20:09:03.093247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:03.093215 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:09:03.248716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:03.248676 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" event={"ID":"f0aeb571-0ee2-4503-bef9-e22ea7dd373c","Type":"ContainerStarted","Data":"24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e"} Nov 01 20:09:03.361233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:03.361126 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" podStartSLOduration=1.5937509730000001 podStartE2EDuration="3.361110727s" totalImagesPullingTime="1.767359754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:03.360059903 +0000 UTC m=+351.034319159" watchObservedRunningTime="2026-11-01 20:09:03.361110727 +0000 UTC m=+351.035369982" Nov 01 20:09:16.155787 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:09:16.155751 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:09:19.232430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:19.232392 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:09:46.155710 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:09:46.155667 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:09:58.325411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:09:58.325368 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:02.260911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:02.260863 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:02.452237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:02.452197 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:16.156863 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:10:16.156818 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:10:18.737791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.737751 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-gljfn"] Nov 01 20:10:18.741483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.741454 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:18.742546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.742511 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:18.795966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.795932 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 20:10:18.797627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.797537 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-gljfn"] Nov 01 20:10:18.836085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.836049 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62j7g\" (UniqueName: \"kubernetes.io/projected/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-kube-api-access-62j7g\") pod \"ai-gateway-controller-7d85675f98-gljfn\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:18.836276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.836178 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-gljfn\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:18.844901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.844875 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 20:10:18.844901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.844879 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-hh5t6\"" Nov 01 20:10:18.937679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.937644 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62j7g\" (UniqueName: \"kubernetes.io/projected/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-kube-api-access-62j7g\") pod \"ai-gateway-controller-7d85675f98-gljfn\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:18.937858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.937761 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-gljfn\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:18.940397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.940373 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-gljfn\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:18.994599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:18.994529 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62j7g\" (UniqueName: \"kubernetes.io/projected/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-kube-api-access-62j7g\") pod \"ai-gateway-controller-7d85675f98-gljfn\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:19.053599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:19.053549 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:19.054309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:19.054291 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:19.424653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:19.424440 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-gljfn"] Nov 01 20:10:19.427558 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:10:19.427525 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bab856_e7c1_4f47_a11a_6bca5e72fc2d.slice/crio-05a73e7d82edfef5990e4c9ffe311931c2c634e24e898ce09192f44f40a48e48 WatchSource:0}: Error finding container 05a73e7d82edfef5990e4c9ffe311931c2c634e24e898ce09192f44f40a48e48: Status 404 returned error can't find the container with id 05a73e7d82edfef5990e4c9ffe311931c2c634e24e898ce09192f44f40a48e48 Nov 01 20:10:19.601441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:19.601402 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:19.625347 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:19.625300 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" event={"ID":"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d","Type":"ContainerStarted","Data":"05a73e7d82edfef5990e4c9ffe311931c2c634e24e898ce09192f44f40a48e48"} Nov 01 20:10:24.654032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:24.653992 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" event={"ID":"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d","Type":"ContainerStarted","Data":"0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8"} Nov 01 20:10:24.654462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:24.654081 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:24.766417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:24.766316 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" podStartSLOduration=2.126782565 podStartE2EDuration="6.766299502s" totalImagesPullingTime="4.639516937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:24.765909691 +0000 UTC m=+432.440168945" watchObservedRunningTime="2026-11-01 20:10:24.766299502 +0000 UTC m=+432.440558758" Nov 01 20:10:27.975512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:27.975472 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:28.215851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.215809 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:28.377428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.377354 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s75g6"] Nov 01 20:10:28.380996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.380976 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:28.438049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.438018 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s75g6"] Nov 01 20:10:28.485078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.485048 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 20:10:28.485249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.485121 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l79xf\"" Nov 01 20:10:28.529663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.529631 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhwb\" (UniqueName: \"kubernetes.io/projected/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-kube-api-access-4dhwb\") pod \"payload-pre-processing-7b98bb678c-s75g6\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:28.529838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.529682 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s75g6\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:28.630690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.630601 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhwb\" (UniqueName: \"kubernetes.io/projected/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-kube-api-access-4dhwb\") pod \"payload-pre-processing-7b98bb678c-s75g6\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:28.630690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.630668 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s75g6\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:28.631426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.631405 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-s75g6\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:28.687438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.687402 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhwb\" (UniqueName: \"kubernetes.io/projected/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-kube-api-access-4dhwb\") pod \"payload-pre-processing-7b98bb678c-s75g6\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:28.694310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:28.694281 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:29.063547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:29.063521 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s75g6"] Nov 01 20:10:29.066369 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:10:29.066317 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb80d241_7c98_4cc5_a861_eb5e8cb2c9fb.slice/crio-37d3acbee1be4919d440fef8576247c30dab21485176fde3d831badeea8e595c WatchSource:0}: Error finding container 37d3acbee1be4919d440fef8576247c30dab21485176fde3d831badeea8e595c: Status 404 returned error can't find the container with id 37d3acbee1be4919d440fef8576247c30dab21485176fde3d831badeea8e595c Nov 01 20:10:29.680798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:29.680767 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" event={"ID":"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb","Type":"ContainerStarted","Data":"37d3acbee1be4919d440fef8576247c30dab21485176fde3d831badeea8e595c"} Nov 01 20:10:32.699754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:32.699710 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" event={"ID":"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb","Type":"ContainerStarted","Data":"e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07"} Nov 01 20:10:32.700263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:32.699861 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:32.812789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:32.812728 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" podStartSLOduration=2.162844313 podStartE2EDuration="4.81271239s" totalImagesPullingTime="2.649868077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:32.811852688 +0000 UTC m=+440.486111945" watchObservedRunningTime="2026-11-01 20:10:32.81271239 +0000 UTC m=+440.486971644" Nov 01 20:10:35.661579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:35.661549 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:36.006582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.006544 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-54d6db5d4f-6rv48"] Nov 01 20:10:36.012175 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.012141 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.063389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.063303 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54d6db5d4f-6rv48"] Nov 01 20:10:36.100071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.100030 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e038c41-6c0d-45d8-8fcd-2c8ac51ba295-metrics-certs\") pod \"ai-gateway-controller-54d6db5d4f-6rv48\" (UID: \"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295\") " pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.100251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.100075 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffs49\" (UniqueName: \"kubernetes.io/projected/2e038c41-6c0d-45d8-8fcd-2c8ac51ba295-kube-api-access-ffs49\") pod \"ai-gateway-controller-54d6db5d4f-6rv48\" (UID: \"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295\") " pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.201154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.201114 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e038c41-6c0d-45d8-8fcd-2c8ac51ba295-metrics-certs\") pod \"ai-gateway-controller-54d6db5d4f-6rv48\" (UID: \"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295\") " pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.201323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.201160 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffs49\" (UniqueName: \"kubernetes.io/projected/2e038c41-6c0d-45d8-8fcd-2c8ac51ba295-kube-api-access-ffs49\") pod \"ai-gateway-controller-54d6db5d4f-6rv48\" (UID: \"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295\") " pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.203825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.203800 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e038c41-6c0d-45d8-8fcd-2c8ac51ba295-metrics-certs\") pod \"ai-gateway-controller-54d6db5d4f-6rv48\" (UID: \"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295\") " pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.258736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.258657 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffs49\" (UniqueName: \"kubernetes.io/projected/2e038c41-6c0d-45d8-8fcd-2c8ac51ba295-kube-api-access-ffs49\") pod \"ai-gateway-controller-54d6db5d4f-6rv48\" (UID: \"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295\") " pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.325493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.325459 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:36.695050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.695027 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54d6db5d4f-6rv48"] Nov 01 20:10:36.697295 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:10:36.697263 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e038c41_6c0d_45d8_8fcd_2c8ac51ba295.slice/crio-76c575e6bd9b43fa2a4ebacbb3269a9f2621b3f3f11ef920f8806778d532e66f WatchSource:0}: Error finding container 76c575e6bd9b43fa2a4ebacbb3269a9f2621b3f3f11ef920f8806778d532e66f: Status 404 returned error can't find the container with id 76c575e6bd9b43fa2a4ebacbb3269a9f2621b3f3f11ef920f8806778d532e66f Nov 01 20:10:36.721305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:36.721270 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" event={"ID":"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295","Type":"ContainerStarted","Data":"76c575e6bd9b43fa2a4ebacbb3269a9f2621b3f3f11ef920f8806778d532e66f"} Nov 01 20:10:37.728056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:37.728019 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" event={"ID":"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295","Type":"ContainerStarted","Data":"5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf"} Nov 01 20:10:37.728572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:37.728146 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:37.841701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:37.841650 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" podStartSLOduration=1.9354684020000001 podStartE2EDuration="2.841635149s" totalImagesPullingTime="906.166747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:37.840365625 +0000 UTC m=+445.514624881" watchObservedRunningTime="2026-11-01 20:10:37.841635149 +0000 UTC m=+445.515894439" Nov 01 20:10:43.708962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:43.708933 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 20:10:46.157370 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:10:46.157314 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:10:47.384931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:47.384892 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:48.735927 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:48.735892 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:10:48.903769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:48.903733 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-gljfn"] Nov 01 20:10:48.904075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:48.903995 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" podUID="d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" containerName="manager" containerID="cri-o://0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8" gracePeriod=30 Nov 01 20:10:49.157025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.157000 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:49.217252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.217215 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-metrics-certs\") pod \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " Nov 01 20:10:49.217455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.217296 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-kube-api-access-62j7g\" (UniqueName: \"kubernetes.io/projected/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-kube-api-access-62j7g\") pod \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\" (UID: \"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d\") " Nov 01 20:10:49.219646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.219611 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-metrics-certs" pod "d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" (UID: "d7bab856-e7c1-4f47-a11a-6bca5e72fc2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:10:49.219646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.219632 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-kube-api-access-62j7g" pod "d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" (UID: "d7bab856-e7c1-4f47-a11a-6bca5e72fc2d"). InnerVolumeSpecName "kube-api-access-62j7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:10:49.318035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.317950 5633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-metrics-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:10:49.318035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.317984 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62j7g\" (UniqueName: \"kubernetes.io/projected/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d-kube-api-access-62j7g\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:10:49.790347 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.790302 5633 generic.go:356] "Generic (PLEG): container finished" podID="d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" containerID="0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8" exitCode=0 Nov 01 20:10:49.790844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.790379 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" Nov 01 20:10:49.790844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.790390 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" event={"ID":"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d","Type":"ContainerDied","Data":"0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8"} Nov 01 20:10:49.790844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.790447 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-gljfn" event={"ID":"d7bab856-e7c1-4f47-a11a-6bca5e72fc2d","Type":"ContainerDied","Data":"05a73e7d82edfef5990e4c9ffe311931c2c634e24e898ce09192f44f40a48e48"} Nov 01 20:10:49.790844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.790473 5633 scope.go:122] "RemoveContainer" containerID="0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8" Nov 01 20:10:49.800524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.800505 5633 scope.go:122] "RemoveContainer" containerID="0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8" Nov 01 20:10:49.800842 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:10:49.800825 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8\": container with ID starting with 0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8 not found: ID does not exist" containerID="0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8" Nov 01 20:10:49.800897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.800852 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8"} err="failed to get container status \"0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8\": rpc error: code = NotFound desc = could not find container \"0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8\": container with ID starting with 0bfeb730c31bc200cb9115683886513b1f239c168786410a721339c84825bfb8 not found: ID does not exist" Nov 01 20:10:49.955253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:49.955216 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-gljfn"] Nov 01 20:10:50.004934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:50.004895 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-gljfn"] Nov 01 20:10:50.421625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:50.421594 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:50.465640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:50.465610 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" path="/var/lib/kubelet/pods/d7bab856-e7c1-4f47-a11a-6bca5e72fc2d/volumes" Nov 01 20:10:50.885387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:50.885320 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:52.039559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:52.039526 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:52.223908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:52.223868 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:10:58.802922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:10:58.802878 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:02.037149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:02.037112 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:11:07.142100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.142056 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m"] Nov 01 20:11:07.142677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.142656 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" containerName="manager" Nov 01 20:11:07.142766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.142681 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" containerName="manager" Nov 01 20:11:07.142846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.142834 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bab856-e7c1-4f47-a11a-6bca5e72fc2d" containerName="manager" Nov 01 20:11:07.150281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.150257 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.204410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.204374 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m"] Nov 01 20:11:07.223970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.223937 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:07.250432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.250403 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 20:11:07.250591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.250437 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 20:11:07.250591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.250518 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wskl6\"" Nov 01 20:11:07.251434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.251408 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 20:11:07.295448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.295404 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6287c3-1095-4e2e-9722-8731aa3a8235-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.295448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.295448 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5cf\" (UniqueName: \"kubernetes.io/projected/ae6287c3-1095-4e2e-9722-8731aa3a8235-kube-api-access-mn5cf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.295699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.295573 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.295699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.295633 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.295811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.295711 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.295811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.295780 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396281 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396367 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396390 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396419 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396492 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6287c3-1095-4e2e-9722-8731aa3a8235-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396516 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5cf\" (UniqueName: \"kubernetes.io/projected/ae6287c3-1095-4e2e-9722-8731aa3a8235-kube-api-access-mn5cf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396883 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.396971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.396925 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.397155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.397129 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.398943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.398916 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6287c3-1095-4e2e-9722-8731aa3a8235-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.400711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.400685 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6287c3-1095-4e2e-9722-8731aa3a8235-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.452876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.452846 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5cf\" (UniqueName: \"kubernetes.io/projected/ae6287c3-1095-4e2e-9722-8731aa3a8235-kube-api-access-mn5cf\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v9p6m\" (UID: \"ae6287c3-1095-4e2e-9722-8731aa3a8235\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.463873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.463841 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:07.837973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.837943 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m"] Nov 01 20:11:07.840088 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:11:07.840060 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6287c3_1095_4e2e_9722_8731aa3a8235.slice/crio-926e6fd0edead7e7b9f1da911c09cd23bd58bb574d564bb9c5e5cae36ff96b13 WatchSource:0}: Error finding container 926e6fd0edead7e7b9f1da911c09cd23bd58bb574d564bb9c5e5cae36ff96b13: Status 404 returned error can't find the container with id 926e6fd0edead7e7b9f1da911c09cd23bd58bb574d564bb9c5e5cae36ff96b13 Nov 01 20:11:07.885400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:07.885363 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" event={"ID":"ae6287c3-1095-4e2e-9722-8731aa3a8235","Type":"ContainerStarted","Data":"926e6fd0edead7e7b9f1da911c09cd23bd58bb574d564bb9c5e5cae36ff96b13"} Nov 01 20:11:09.460276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:09.460228 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:11:09.899253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:09.899211 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" event={"ID":"ae6287c3-1095-4e2e-9722-8731aa3a8235","Type":"ContainerStarted","Data":"360723b5d47ada4b2bfd1f8390cbd4ac05b36b795b0ee4c32fffb58faf963e2e"} Nov 01 20:11:10.015904 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.015848 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" podStartSLOduration=1.51558115 podStartE2EDuration="3.015832879s" totalImagesPullingTime="1.500251729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:10.014490859 +0000 UTC m=+477.688750115" watchObservedRunningTime="2026-11-01 20:11:10.015832879 +0000 UTC m=+477.690092133" Nov 01 20:11:10.465203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.465175 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:10.664240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.664208 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h"] Nov 01 20:11:10.668162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.668136 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.720508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.720412 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 20:11:10.732789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.732755 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd4b73a-baf7-471d-8129-73c6934027ee-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.732972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.732796 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.732972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.732906 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.733085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.732995 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.733152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.733129 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.733209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.733171 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlbv\" (UniqueName: \"kubernetes.io/projected/fcd4b73a-baf7-471d-8129-73c6934027ee-kube-api-access-5tlbv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.747717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.747681 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:10.771480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.771446 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h"] Nov 01 20:11:10.833959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.833923 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.833981 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.834095 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlbv\" (UniqueName: \"kubernetes.io/projected/fcd4b73a-baf7-471d-8129-73c6934027ee-kube-api-access-5tlbv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.834219 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd4b73a-baf7-471d-8129-73c6934027ee-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.834252 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.834293 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.834451 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.834477 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.834633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.834585 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.836733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.836707 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcd4b73a-baf7-471d-8129-73c6934027ee-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.836935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.836916 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd4b73a-baf7-471d-8129-73c6934027ee-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.891269 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.891226 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlbv\" (UniqueName: \"kubernetes.io/projected/fcd4b73a-baf7-471d-8129-73c6934027ee-kube-api-access-5tlbv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vkc6h\" (UID: \"fcd4b73a-baf7-471d-8129-73c6934027ee\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:10.981821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:10.981733 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:11.355260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:11.355202 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h"] Nov 01 20:11:11.357537 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:11:11.357499 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd4b73a_baf7_471d_8129_73c6934027ee.slice/crio-47a56dc3aae45e42ccf567541522a72a6379bb18017670448ebecaac7b193377 WatchSource:0}: Error finding container 47a56dc3aae45e42ccf567541522a72a6379bb18017670448ebecaac7b193377: Status 404 returned error can't find the container with id 47a56dc3aae45e42ccf567541522a72a6379bb18017670448ebecaac7b193377 Nov 01 20:11:11.911608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:11.911571 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" event={"ID":"fcd4b73a-baf7-471d-8129-73c6934027ee","Type":"ContainerStarted","Data":"47a56dc3aae45e42ccf567541522a72a6379bb18017670448ebecaac7b193377"} Nov 01 20:11:12.918672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:12.918578 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" event={"ID":"fcd4b73a-baf7-471d-8129-73c6934027ee","Type":"ContainerStarted","Data":"51e647ad91b6c25a08f3f4ae31cc4815671514858db79033b1c6077cc3d065f5"} Nov 01 20:11:12.982692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:12.982655 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:13.036145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:13.036095 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" podStartSLOduration=2.316726391 podStartE2EDuration="3.036079086s" totalImagesPullingTime="719.352695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:13.033770239 +0000 UTC m=+480.708029495" watchObservedRunningTime="2026-11-01 20:11:13.036079086 +0000 UTC m=+480.710338341" Nov 01 20:11:15.080903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.080863 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn"] Nov 01 20:11:15.085801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.085773 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.164795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.164754 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:15.175760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.175730 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9vm\" (UniqueName: \"kubernetes.io/projected/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-kube-api-access-hs9vm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.175916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.175773 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.175916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.175868 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.176061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.175926 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.176061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.176041 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.176172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.176095 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.189543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.189518 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 01 20:11:15.192540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.192507 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn"] Nov 01 20:11:15.276808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.276752 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.276871 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.276953 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.276985 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.277014 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9vm\" (UniqueName: \"kubernetes.io/projected/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-kube-api-access-hs9vm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.277051 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.277362 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.277398 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.277783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.277755 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.279939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.279912 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.280280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.280260 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.341090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.340109 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9vm\" (UniqueName: \"kubernetes.io/projected/dbc07a3f-9280-4bf8-aa49-b59a3a4383a9-kube-api-access-hs9vm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jgzgn\" (UID: \"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.400545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.400503 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:15.792153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.792125 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn"] Nov 01 20:11:15.794132 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:11:15.794104 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc07a3f_9280_4bf8_aa49_b59a3a4383a9.slice/crio-6b6b60b2842e43fb0f4c72004b63c4a61f0ddcb6fefef1c8191820498b82bdab WatchSource:0}: Error finding container 6b6b60b2842e43fb0f4c72004b63c4a61f0ddcb6fefef1c8191820498b82bdab: Status 404 returned error can't find the container with id 6b6b60b2842e43fb0f4c72004b63c4a61f0ddcb6fefef1c8191820498b82bdab Nov 01 20:11:15.935249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:15.935207 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" event={"ID":"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9","Type":"ContainerStarted","Data":"6b6b60b2842e43fb0f4c72004b63c4a61f0ddcb6fefef1c8191820498b82bdab"} Nov 01 20:11:16.157631 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:11:16.157533 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:11:16.942492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:16.942402 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" event={"ID":"dbc07a3f-9280-4bf8-aa49-b59a3a4383a9","Type":"ContainerStarted","Data":"20e63169571d75e6a1557e83bf450be3501ff45fb40970664c3b59fa6285961a"} Nov 01 20:11:17.058648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:17.058592 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" podStartSLOduration=1.297205548 podStartE2EDuration="2.058576303s" totalImagesPullingTime="761.370755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:17.057175122 +0000 UTC m=+484.731434377" watchObservedRunningTime="2026-11-01 20:11:17.058576303 +0000 UTC m=+484.732835580" Nov 01 20:11:17.401225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:17.401176 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:17.464803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:17.464769 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:17.477417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:17.477392 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:17.959359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:17.959305 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v9p6m" Nov 01 20:11:19.080723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:19.080691 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:19.168563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:19.168529 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:11:19.910026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:19.909986 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:20.982549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:20.982505 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:20.995187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:20.995159 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:21.981197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:21.981167 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vkc6h" Nov 01 20:11:22.209118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.209081 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4v5f2"] Nov 01 20:11:22.212957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.212936 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.265494 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.265415 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 20:11:22.271795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.271766 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4v5f2"] Nov 01 20:11:22.289630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.289600 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:22.356231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.356199 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.356453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.356238 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.356453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.356260 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jcxf\" (UniqueName: \"kubernetes.io/projected/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-kube-api-access-4jcxf\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.356453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.356375 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.356578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.356542 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.356578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.356571 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-home\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.457917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.457868 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.457917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.457923 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.458176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.457947 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jcxf\" (UniqueName: \"kubernetes.io/projected/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-kube-api-access-4jcxf\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.458176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.457978 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.458176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.458093 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.458176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.458123 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-home\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.458521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.458492 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.458634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.458549 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.458634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.458596 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-home\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.460747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.460715 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.461010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.460965 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.515414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.515381 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jcxf\" (UniqueName: \"kubernetes.io/projected/c299f460-9ff7-4b2d-ba42-50d6d4a5e810-kube-api-access-4jcxf\") pod \"echo-simulator-kserve-64b6f8f96c-4v5f2\" (UID: \"c299f460-9ff7-4b2d-ba42-50d6d4a5e810\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.526359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.526266 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:22.897565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.897538 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4v5f2"] Nov 01 20:11:22.899707 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:11:22.899678 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc299f460_9ff7_4b2d_ba42_50d6d4a5e810.slice/crio-6004c46072553f40a7cd76fbe32bb8400bb305992c7a2115dc89da2ecd7828b3 WatchSource:0}: Error finding container 6004c46072553f40a7cd76fbe32bb8400bb305992c7a2115dc89da2ecd7828b3: Status 404 returned error can't find the container with id 6004c46072553f40a7cd76fbe32bb8400bb305992c7a2115dc89da2ecd7828b3 Nov 01 20:11:22.976779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:22.976732 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" event={"ID":"c299f460-9ff7-4b2d-ba42-50d6d4a5e810","Type":"ContainerStarted","Data":"6004c46072553f40a7cd76fbe32bb8400bb305992c7a2115dc89da2ecd7828b3"} Nov 01 20:11:25.401042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:25.400991 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:25.416361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:25.416270 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:25.995562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:25.995508 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" event={"ID":"c299f460-9ff7-4b2d-ba42-50d6d4a5e810","Type":"ContainerStarted","Data":"cad4f9daebc7da648e20bbdad66c12bec3bb3561929a010d1d2d0026f20b51ea"} Nov 01 20:11:26.007075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:26.007045 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jgzgn" Nov 01 20:11:26.112026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:26.111971 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" podStartSLOduration=1.4639045560000001 podStartE2EDuration="4.111949922s" totalImagesPullingTime="2.648045366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:26.110870914 +0000 UTC m=+493.785130173" watchObservedRunningTime="2026-11-01 20:11:26.111949922 +0000 UTC m=+493.786209177" Nov 01 20:11:26.526662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:26.526619 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:27.059854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:27.059822 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:30.101957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:30.101918 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:32.527346 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:32.527285 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:32.540630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:32.540601 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:32.975496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:32.975466 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:11:33.045283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:33.045241 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4v5f2" Nov 01 20:11:39.149221 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.149182 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq"] Nov 01 20:11:39.153138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.153119 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.204833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.204802 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 20:11:39.204833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.204837 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\"" Nov 01 20:11:39.222319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222281 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222421 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222468 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222574 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkkb\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-kube-api-access-pwkkb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222609 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222649 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222703 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222751 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1710eb6f-7d43-4a4f-9d6a-799369f80228-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.222783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.222771 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.257225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.257186 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq"] Nov 01 20:11:39.323787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323749 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkkb\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-kube-api-access-pwkkb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.323787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323801 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323831 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323849 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323868 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1710eb6f-7d43-4a4f-9d6a-799369f80228-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323896 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323929 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.323980 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.324006 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.324311 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.324529 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.324561 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.324586 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.324792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.324727 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1710eb6f-7d43-4a4f-9d6a-799369f80228-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.326361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.326321 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.326583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.326565 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.380536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.380506 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.380862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.380836 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkkb\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-kube-api-access-pwkkb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bp6cq\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.466136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.466109 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:39.941516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:39.941492 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq"] Nov 01 20:11:39.943159 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:11:39.943130 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1710eb6f_7d43_4a4f_9d6a_799369f80228.slice/crio-e909e837792cba52b831ae884db1b0afd53e1622ddd14af62444f9c118e25904 WatchSource:0}: Error finding container e909e837792cba52b831ae884db1b0afd53e1622ddd14af62444f9c118e25904: Status 404 returned error can't find the container with id e909e837792cba52b831ae884db1b0afd53e1622ddd14af62444f9c118e25904 Nov 01 20:11:40.071967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:40.071932 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" event={"ID":"1710eb6f-7d43-4a4f-9d6a-799369f80228","Type":"ContainerStarted","Data":"e909e837792cba52b831ae884db1b0afd53e1622ddd14af62444f9c118e25904"} Nov 01 20:11:44.158202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:44.158167 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:11:44.158851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:44.158251 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:11:44.158851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:44.158361 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:11:45.114602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:45.114548 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" event={"ID":"1710eb6f-7d43-4a4f-9d6a-799369f80228","Type":"ContainerStarted","Data":"be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a"} Nov 01 20:11:45.467165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:45.467131 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:45.468447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:45.468422 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:11:45.468511 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:45.468472 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 20:11:46.155846 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:11:46.155808 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:11:46.466504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:46.466475 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:11:46.466720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:46.466525 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 20:11:47.466927 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:47.466895 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:11:47.467416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:47.466951 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 20:11:48.468956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:48.468919 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:11:48.469370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:48.468990 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 20:11:49.466601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:49.466564 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:11:49.466808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:49.466643 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 20:11:50.466631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:50.466595 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:11:50.467096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:50.466653 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 20:11:51.466745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:51.466705 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:11:51.467147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:51.466778 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 01 20:11:52.681250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:52.681203 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:53.493211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:53.493169 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bp6cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:11:53.493420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:53.493245 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:53.500036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:53.499994 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:53.500470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:53.500446 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:53.501253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:53.501232 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:11:53.668402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:53.668347 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podStartSLOduration=10.456434849 podStartE2EDuration="14.668313474s" totalImagesPullingTime="4.211878625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:45.329406095 +0000 UTC m=+513.003665348" watchObservedRunningTime="2026-11-01 20:11:53.668313474 +0000 UTC m=+521.342572729" Nov 01 20:11:55.835064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:55.835023 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cf654df49-4ch47"] Nov 01 20:11:55.838853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:55.838832 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:55.894067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:55.894034 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cf654df49-4ch47"] Nov 01 20:11:56.016204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.016157 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1ef9e76-0e8b-4050-b269-5042540f7826-tls-cert\") pod \"authorino-5cf654df49-4ch47\" (UID: \"f1ef9e76-0e8b-4050-b269-5042540f7826\") " pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:56.016204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.016207 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6q2d\" (UniqueName: \"kubernetes.io/projected/f1ef9e76-0e8b-4050-b269-5042540f7826-kube-api-access-l6q2d\") pod \"authorino-5cf654df49-4ch47\" (UID: \"f1ef9e76-0e8b-4050-b269-5042540f7826\") " pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:56.117294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.117203 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1ef9e76-0e8b-4050-b269-5042540f7826-tls-cert\") pod \"authorino-5cf654df49-4ch47\" (UID: \"f1ef9e76-0e8b-4050-b269-5042540f7826\") " pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:56.117294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.117240 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q2d\" (UniqueName: \"kubernetes.io/projected/f1ef9e76-0e8b-4050-b269-5042540f7826-kube-api-access-l6q2d\") pod \"authorino-5cf654df49-4ch47\" (UID: \"f1ef9e76-0e8b-4050-b269-5042540f7826\") " pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:56.119965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.119942 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1ef9e76-0e8b-4050-b269-5042540f7826-tls-cert\") pod \"authorino-5cf654df49-4ch47\" (UID: \"f1ef9e76-0e8b-4050-b269-5042540f7826\") " pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:56.174522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.174489 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6q2d\" (UniqueName: \"kubernetes.io/projected/f1ef9e76-0e8b-4050-b269-5042540f7826-kube-api-access-l6q2d\") pod \"authorino-5cf654df49-4ch47\" (UID: \"f1ef9e76-0e8b-4050-b269-5042540f7826\") " pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:56.383251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.383168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:56.450520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.450482 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cf654df49-4ch47" Nov 01 20:11:56.523938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.523892 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:56.823804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:56.823780 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cf654df49-4ch47"] Nov 01 20:11:56.825922 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:11:56.825891 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1ef9e76_0e8b_4050_b269_5042540f7826.slice/crio-d8c665e01f1e577be94641c1bafd0982e8363f1a56f532b704508e838053303c WatchSource:0}: Error finding container d8c665e01f1e577be94641c1bafd0982e8363f1a56f532b704508e838053303c: Status 404 returned error can't find the container with id d8c665e01f1e577be94641c1bafd0982e8363f1a56f532b704508e838053303c Nov 01 20:11:57.183001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:57.182922 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cf654df49-4ch47" event={"ID":"f1ef9e76-0e8b-4050-b269-5042540f7826","Type":"ContainerStarted","Data":"d8c665e01f1e577be94641c1bafd0982e8363f1a56f532b704508e838053303c"} Nov 01 20:11:57.888767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:57.888732 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:58.025565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:58.025523 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:11:59.196057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.196019 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cf654df49-4ch47" event={"ID":"f1ef9e76-0e8b-4050-b269-5042540f7826","Type":"ContainerStarted","Data":"b323fac17a7e7bf9c86ed7b0508cf29b3ef043bfafd206b3d4027eb996bcbf4c"} Nov 01 20:11:59.320454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.320385 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5cf654df49-4ch47" podStartSLOduration=2.742983796 podStartE2EDuration="4.320362391s" totalImagesPullingTime="1.577378595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:59.314026513 +0000 UTC m=+526.988285769" watchObservedRunningTime="2026-11-01 20:11:59.320362391 +0000 UTC m=+526.994621647" Nov 01 20:11:59.380385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.380349 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bb6ff9d46-d6s2q"] Nov 01 20:11:59.380617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.380580 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" podUID="f0aeb571-0ee2-4503-bef9-e22ea7dd373c" containerName="authorino" containerID="cri-o://24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e" gracePeriod=30 Nov 01 20:11:59.646519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.646488 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:11:59.758594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.758505 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-kube-api-access-67ckm\" (UniqueName: \"kubernetes.io/projected/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-kube-api-access-67ckm\") pod \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " Nov 01 20:11:59.758594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.758577 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-tls-cert\" (UniqueName: \"kubernetes.io/secret/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-tls-cert\") pod \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\" (UID: \"f0aeb571-0ee2-4503-bef9-e22ea7dd373c\") " Nov 01 20:11:59.761209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.761169 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-kube-api-access-67ckm" pod "f0aeb571-0ee2-4503-bef9-e22ea7dd373c" (UID: "f0aeb571-0ee2-4503-bef9-e22ea7dd373c"). InnerVolumeSpecName "kube-api-access-67ckm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:11:59.772890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.772851 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-tls-cert" pod "f0aeb571-0ee2-4503-bef9-e22ea7dd373c" (UID: "f0aeb571-0ee2-4503-bef9-e22ea7dd373c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:11:59.862370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.860152 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67ckm\" (UniqueName: \"kubernetes.io/projected/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-kube-api-access-67ckm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:11:59.862370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:11:59.860199 5633 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0aeb571-0ee2-4503-bef9-e22ea7dd373c-tls-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:12:00.204867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.204733 5633 generic.go:356] "Generic (PLEG): container finished" podID="f0aeb571-0ee2-4503-bef9-e22ea7dd373c" containerID="24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e" exitCode=0 Nov 01 20:12:00.204867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.204836 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" Nov 01 20:12:00.224135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.204921 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" event={"ID":"f0aeb571-0ee2-4503-bef9-e22ea7dd373c","Type":"ContainerDied","Data":"24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e"} Nov 01 20:12:00.224135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.204977 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb6ff9d46-d6s2q" event={"ID":"f0aeb571-0ee2-4503-bef9-e22ea7dd373c","Type":"ContainerDied","Data":"fa7c772c80c293bfd615d7b596078cdf997d6a8e1327b82d93d864ab0f194f7e"} Nov 01 20:12:00.224135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.205002 5633 scope.go:122] "RemoveContainer" containerID="24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e" Nov 01 20:12:00.224135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.216807 5633 scope.go:122] "RemoveContainer" containerID="24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e" Nov 01 20:12:00.224135 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:12:00.217122 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e\": container with ID starting with 24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e not found: ID does not exist" containerID="24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e" Nov 01 20:12:00.224135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.217150 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e"} err="failed to get container status \"24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e\": rpc error: code = NotFound desc = could not find container \"24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e\": container with ID starting with 24c8a481c423826e37ef6fcfab849831184a276e739b17cabb51c7c1b400de3e not found: ID does not exist" Nov 01 20:12:00.372040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.372012 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bb6ff9d46-d6s2q"] Nov 01 20:12:00.375394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.375367 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bb6ff9d46-d6s2q"] Nov 01 20:12:00.466802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:00.466768 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0aeb571-0ee2-4503-bef9-e22ea7dd373c" path="/var/lib/kubelet/pods/f0aeb571-0ee2-4503-bef9-e22ea7dd373c/volumes" Nov 01 20:12:16.155816 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:12:16.155773 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:12:20.256312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:20.256248 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:28.221067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.220971 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:28.532245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.532154 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:28.755627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.755592 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w"] Nov 01 20:12:28.756137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.756117 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f0aeb571-0ee2-4503-bef9-e22ea7dd373c" containerName="authorino" Nov 01 20:12:28.756243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.756138 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0aeb571-0ee2-4503-bef9-e22ea7dd373c" containerName="authorino" Nov 01 20:12:28.756298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.756261 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0aeb571-0ee2-4503-bef9-e22ea7dd373c" containerName="authorino" Nov 01 20:12:28.759793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.759768 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.811656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.811579 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\"" Nov 01 20:12:28.847416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.847384 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:28.854510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854480 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk249\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-kube-api-access-dk249\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854524 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854545 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854678 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5696e679-5d84-4352-8122-42262f1457e0-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854741 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854808 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854871 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854934 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.854988 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.854974 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5696e679-5d84-4352-8122-42262f1457e0-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.863233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.863208 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w"] Nov 01 20:12:28.956048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956013 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956057 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956087 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956135 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956178 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5696e679-5d84-4352-8122-42262f1457e0-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956203 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk249\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-kube-api-access-dk249\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956241 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956269 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956316 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5696e679-5d84-4352-8122-42262f1457e0-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956601 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956636 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.956888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956859 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.957005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.956966 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.957056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.957038 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5696e679-5d84-4352-8122-42262f1457e0-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.958824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.958799 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:28.958931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:28.958912 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5696e679-5d84-4352-8122-42262f1457e0-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:29.013042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.013009 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:29.013233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.013197 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk249\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-kube-api-access-dk249\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:29.033121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.033088 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:29.060859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.060825 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v"] Nov 01 20:12:29.065882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.065821 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.072663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.072637 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:29.126207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.126168 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v"] Nov 01 20:12:29.139054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.138743 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1b1134-openshift-default-dockercfg-pvzt4\"" Nov 01 20:12:29.158515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.158478 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.158680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.158624 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.158800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.158782 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.158865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.158828 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28fn\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-kube-api-access-h28fn\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.158921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.158906 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.159018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.158998 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.159096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.159034 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.159096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.159061 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.159195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.159100 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.176047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.175753 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:29.260139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260102 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260152 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260199 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260217 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28fn\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-kube-api-access-h28fn\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260247 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260417 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260473 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260507 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.260666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260552 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.261102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260734 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.261102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260776 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.261102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260902 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.261102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260960 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.261102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.260998 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.262874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.262852 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.262958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.262918 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.322609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.322534 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28fn\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-kube-api-access-h28fn\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.323969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.323942 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.380795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.380756 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:29.381660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.381634 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:29.510620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.510593 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w"] Nov 01 20:12:29.512542 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:12:29.512510 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5696e679_5d84_4352_8122_42262f1457e0.slice/crio-fbfe5e26056bd645970fde2e83bb6a8590814bb8762a7523ad25b054391bfdc9 WatchSource:0}: Error finding container fbfe5e26056bd645970fde2e83bb6a8590814bb8762a7523ad25b054391bfdc9: Status 404 returned error can't find the container with id fbfe5e26056bd645970fde2e83bb6a8590814bb8762a7523ad25b054391bfdc9 Nov 01 20:12:29.514812 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.514778 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:29.514905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.514859 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:29.514972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.514902 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:29.762821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.762793 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v"] Nov 01 20:12:29.764890 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:12:29.764853 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ccf9e0_4a1f_44e9_b032_13d7d763dbdd.slice/crio-7746138cefd08aeb1b7196cd035405f99438ddebc06cafa668789d589c364b49 WatchSource:0}: Error finding container 7746138cefd08aeb1b7196cd035405f99438ddebc06cafa668789d589c364b49: Status 404 returned error can't find the container with id 7746138cefd08aeb1b7196cd035405f99438ddebc06cafa668789d589c364b49 Nov 01 20:12:29.767241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.767203 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:29.767319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.767293 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:29.767413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.767396 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:29.945304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.945271 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k"] Nov 01 20:12:29.949021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:29.949001 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.012654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.012622 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k"] Nov 01 20:12:30.030660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.030624 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:30.062446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.062415 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-381df3-openshift-default-dockercfg-mqqw5\"" Nov 01 20:12:30.068144 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.068112 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-podinfo\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.068312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.068159 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-socket\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.068312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.068202 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-data\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.068467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.068428 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-envoy\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.068531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.068505 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997fw\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-kube-api-access-997fw\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.068587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.068563 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istiod-ca-cert\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.069093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.069065 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-certs\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.069480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.069128 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-credential-socket\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.069480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.069209 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-token\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.170946 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-podinfo\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.170993 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-socket\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171023 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-data\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171054 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-envoy\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171079 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997fw\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-kube-api-access-997fw\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171097 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istiod-ca-cert\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171145 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-certs\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171170 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-credential-socket\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171201 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-token\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171650 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-data\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171742 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171713 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-socket\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171792 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-certs\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.171874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.171835 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-credential-socket\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.172248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.172191 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istiod-ca-cert\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.173827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.173800 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-envoy\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.174098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.174080 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-podinfo\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.187616 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.187485 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:30.227912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.227881 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-token\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.228099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.228074 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-997fw\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-kube-api-access-997fw\") pod \"e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.263835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.263802 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:30.380813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.380767 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" event={"ID":"5696e679-5d84-4352-8122-42262f1457e0","Type":"ContainerStarted","Data":"215a2af3ae678256d293364fb9e6496c5a3fb45ef7f8e943c92bfb160b2fc0e5"} Nov 01 20:12:30.381003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.380844 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" event={"ID":"5696e679-5d84-4352-8122-42262f1457e0","Type":"ContainerStarted","Data":"fbfe5e26056bd645970fde2e83bb6a8590814bb8762a7523ad25b054391bfdc9"} Nov 01 20:12:30.382834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.382802 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" event={"ID":"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd","Type":"ContainerStarted","Data":"c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955"} Nov 01 20:12:30.382968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.382867 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" event={"ID":"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd","Type":"ContainerStarted","Data":"7746138cefd08aeb1b7196cd035405f99438ddebc06cafa668789d589c364b49"} Nov 01 20:12:30.497526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.497479 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podStartSLOduration=2.497465144 podStartE2EDuration="2.497465144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:30.496241586 +0000 UTC m=+558.170500840" watchObservedRunningTime="2026-11-01 20:12:30.497465144 +0000 UTC m=+558.171724399" Nov 01 20:12:30.616204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.616149 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podStartSLOduration=1.6161312429999999 podStartE2EDuration="1.616131243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:30.613297053 +0000 UTC m=+558.287556307" watchObservedRunningTime="2026-11-01 20:12:30.616131243 +0000 UTC m=+558.290390495" Nov 01 20:12:30.641238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.641212 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k"] Nov 01 20:12:30.643278 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:12:30.643235 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad5c1c6_a938_4cd0_98f9_9efb5158aa64.slice/crio-3a3af0f57579cecdeda3f1045efacf79dd22b8164bd6ba20bf9ab0447373f053 WatchSource:0}: Error finding container 3a3af0f57579cecdeda3f1045efacf79dd22b8164bd6ba20bf9ab0447373f053: Status 404 returned error can't find the container with id 3a3af0f57579cecdeda3f1045efacf79dd22b8164bd6ba20bf9ab0447373f053 Nov 01 20:12:30.645684 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.645652 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:30.645761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.645716 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:30.645761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:30.645748 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:31.073498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.073461 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:31.079671 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.079640 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:31.382677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.382589 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:31.388851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.388823 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:31.391613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.391582 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" event={"ID":"fad5c1c6-a938-4cd0-98f9-9efb5158aa64","Type":"ContainerStarted","Data":"a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b"} Nov 01 20:12:31.391779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.391632 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" event={"ID":"fad5c1c6-a938-4cd0-98f9-9efb5158aa64","Type":"ContainerStarted","Data":"3a3af0f57579cecdeda3f1045efacf79dd22b8164bd6ba20bf9ab0447373f053"} Nov 01 20:12:31.392120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.392098 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:31.392238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.392148 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:31.392861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.392833 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:12:31.392955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.392915 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:12:31.741987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:31.741929 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podStartSLOduration=2.741914351 podStartE2EDuration="2.741914351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:31.741456821 +0000 UTC m=+559.415716069" watchObservedRunningTime="2026-11-01 20:12:31.741914351 +0000 UTC m=+559.416173605" Nov 01 20:12:32.264531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:32.264484 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:32.269888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:32.269863 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:32.398258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:32.398226 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:32.399096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:32.399076 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:12:33.310264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:33.310224 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:33.478569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:33.478533 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:33.609119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:33.609018 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:34.099699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:34.099662 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:34.463030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:34.462978 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:35.952306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:35.952254 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:36.133537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:36.133485 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:36.261026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:36.260986 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:36.533650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:36.533553 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:36.703976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:36.703937 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:39.826936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:39.826899 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:39.976624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:39.976588 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:40.139946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:40.139846 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:41.267489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:41.267454 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:41.445364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:41.445318 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:43.693856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:43.693820 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:43.861168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:43.861122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:44.631154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:44.631122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:44.789795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:44.789754 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:44.920835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:44.920687 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:46.156752 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:12:46.156716 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:12:47.880678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:47.880639 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:12:49.561758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:49.561715 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:49.738395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:49.738360 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:49.862095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:49.861999 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:51.235346 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:51.235304 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:54.613275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:54.613180 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:12:56.417786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.417741 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:12:56.784638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.784591 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj"] Nov 01 20:12:56.789723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.789691 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.853846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.853816 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:12:56.894636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.894592 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.894909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.894662 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fhd\" (UniqueName: \"kubernetes.io/projected/180210f7-1850-414d-9c2e-5fb3f07f07ee-kube-api-access-p2fhd\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.894909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.894792 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.899691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.899633 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:12:56.899691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.899650 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:12:56.900126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.900102 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:12:56.914190 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.914152 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj"] Nov 01 20:12:56.995989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.995937 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.996203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.996012 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fhd\" (UniqueName: \"kubernetes.io/projected/180210f7-1850-414d-9c2e-5fb3f07f07ee-kube-api-access-p2fhd\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.996203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.996154 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.997022 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.996966 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:56.999828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:56.999790 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:57.054602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:57.054512 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fhd\" (UniqueName: \"kubernetes.io/projected/180210f7-1850-414d-9c2e-5fb3f07f07ee-kube-api-access-p2fhd\") pod \"maas-api-revoke-sub-24416a5c-5knsj\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:57.104510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:57.104459 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:12:57.530991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:57.530957 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj"] Nov 01 20:12:57.533347 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:12:57.533277 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180210f7_1850_414d_9c2e_5fb3f07f07ee.slice/crio-fb8bbec525ef2fe80358c204eebb3a3c0bdd43ffad93552b5041392c9797a9bc WatchSource:0}: Error finding container fb8bbec525ef2fe80358c204eebb3a3c0bdd43ffad93552b5041392c9797a9bc: Status 404 returned error can't find the container with id fb8bbec525ef2fe80358c204eebb3a3c0bdd43ffad93552b5041392c9797a9bc Nov 01 20:12:57.565462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:57.565418 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" event={"ID":"180210f7-1850-414d-9c2e-5fb3f07f07ee","Type":"ContainerStarted","Data":"fb8bbec525ef2fe80358c204eebb3a3c0bdd43ffad93552b5041392c9797a9bc"} Nov 01 20:12:58.575951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:58.575919 5633 generic.go:356] "Generic (PLEG): container finished" podID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerID="c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955" exitCode=0 Nov 01 20:12:58.576421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:58.575982 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" event={"ID":"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd","Type":"ContainerDied","Data":"c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955"} Nov 01 20:12:58.576421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:58.576390 5633 scope.go:122] "RemoveContainer" containerID="c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955" Nov 01 20:12:58.577656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:58.577616 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:58.577738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:58.577686 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:58.577738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:58.577719 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:12:59.605356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:59.604784 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" event={"ID":"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd","Type":"ContainerStarted","Data":"89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786"} Nov 01 20:12:59.685377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:12:59.685102 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:00.403357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:00.401696 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:13:00.403357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:00.401847 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:13:00.403357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:00.401894 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 20:13:01.383359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:01.382652 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:13:01.383359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:01.382723 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 20:13:01.395358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:01.393459 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:13:02.382381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:02.382314 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:13:02.382566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:02.382410 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 20:13:03.382738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:03.382701 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:13:03.383125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:03.382775 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 20:13:04.406240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:04.405757 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:13:04.406240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:04.405823 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 20:13:04.938976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:04.938939 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:05.382889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:05.382787 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:13:05.382889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:05.382857 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 20:13:05.722164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:05.722123 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" event={"ID":"180210f7-1850-414d-9c2e-5fb3f07f07ee","Type":"ContainerStarted","Data":"e3fbfd23b2cc065893d22b5fbe6d8bbd4dc9054f42bed31f0aa9ed05492ae949"} Nov 01 20:13:05.835901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:05.835843 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" podStartSLOduration=2.510961685 podStartE2EDuration="9.835829466s" totalImagesPullingTime="7.324867781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:05.835236428 +0000 UTC m=+593.509495684" watchObservedRunningTime="2026-11-01 20:13:05.835829466 +0000 UTC m=+593.510088721" Nov 01 20:13:06.387920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:06.387826 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:13:06.389161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:06.389138 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:13:06.733312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:06.733269 5633 generic.go:356] "Generic (PLEG): container finished" podID="180210f7-1850-414d-9c2e-5fb3f07f07ee" containerID="e3fbfd23b2cc065893d22b5fbe6d8bbd4dc9054f42bed31f0aa9ed05492ae949" exitCode=0 Nov 01 20:13:06.734097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:06.734051 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" event={"ID":"180210f7-1850-414d-9c2e-5fb3f07f07ee","Type":"ContainerDied","Data":"e3fbfd23b2cc065893d22b5fbe6d8bbd4dc9054f42bed31f0aa9ed05492ae949"} Nov 01 20:13:07.741448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.741403 5633 generic.go:356] "Generic (PLEG): container finished" podID="5696e679-5d84-4352-8122-42262f1457e0" containerID="215a2af3ae678256d293364fb9e6496c5a3fb45ef7f8e943c92bfb160b2fc0e5" exitCode=0 Nov 01 20:13:07.741952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.741449 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" event={"ID":"5696e679-5d84-4352-8122-42262f1457e0","Type":"ContainerDied","Data":"215a2af3ae678256d293364fb9e6496c5a3fb45ef7f8e943c92bfb160b2fc0e5"} Nov 01 20:13:07.742026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.741959 5633 scope.go:122] "RemoveContainer" containerID="215a2af3ae678256d293364fb9e6496c5a3fb45ef7f8e943c92bfb160b2fc0e5" Nov 01 20:13:07.743485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.743450 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:07.743597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.743534 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:07.743597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.743551 5633 generic.go:356] "Generic (PLEG): container finished" podID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerID="a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b" exitCode=0 Nov 01 20:13:07.743597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.743570 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:07.743698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.743623 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" event={"ID":"fad5c1c6-a938-4cd0-98f9-9efb5158aa64","Type":"ContainerDied","Data":"a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b"} Nov 01 20:13:07.744055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.743995 5633 scope.go:122] "RemoveContainer" containerID="a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b" Nov 01 20:13:07.745546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.745517 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:07.745661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.745642 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:07.745716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.745700 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:07.904840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:07.904813 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:13:08.006147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.005995 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-xks-ca\") pod \"180210f7-1850-414d-9c2e-5fb3f07f07ee\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " Nov 01 20:13:08.006382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.006210 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-openshift-ca\") pod \"180210f7-1850-414d-9c2e-5fb3f07f07ee\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " Nov 01 20:13:08.006382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.006287 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180210f7-1850-414d-9c2e-5fb3f07f07ee-kube-api-access-p2fhd\" (UniqueName: \"kubernetes.io/projected/180210f7-1850-414d-9c2e-5fb3f07f07ee-kube-api-access-p2fhd\") pod \"180210f7-1850-414d-9c2e-5fb3f07f07ee\" (UID: \"180210f7-1850-414d-9c2e-5fb3f07f07ee\") " Nov 01 20:13:08.006617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.006590 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-openshift-ca" pod "180210f7-1850-414d-9c2e-5fb3f07f07ee" (UID: "180210f7-1850-414d-9c2e-5fb3f07f07ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:13:08.006911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.006689 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:13:08.008934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.008907 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180210f7-1850-414d-9c2e-5fb3f07f07ee-kube-api-access-p2fhd" pod "180210f7-1850-414d-9c2e-5fb3f07f07ee" (UID: "180210f7-1850-414d-9c2e-5fb3f07f07ee"). InnerVolumeSpecName "kube-api-access-p2fhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:13:08.009135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.009108 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-xks-ca" pod "180210f7-1850-414d-9c2e-5fb3f07f07ee" (UID: "180210f7-1850-414d-9c2e-5fb3f07f07ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:13:08.107551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.107517 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180210f7-1850-414d-9c2e-5fb3f07f07ee-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:13:08.107551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.107548 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2fhd\" (UniqueName: \"kubernetes.io/projected/180210f7-1850-414d-9c2e-5fb3f07f07ee-kube-api-access-p2fhd\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:13:08.521437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.519181 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:08.752361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.752118 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" event={"ID":"fad5c1c6-a938-4cd0-98f9-9efb5158aa64","Type":"ContainerStarted","Data":"51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5"} Nov 01 20:13:08.753532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.753508 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" event={"ID":"180210f7-1850-414d-9c2e-5fb3f07f07ee","Type":"ContainerDied","Data":"fb8bbec525ef2fe80358c204eebb3a3c0bdd43ffad93552b5041392c9797a9bc"} Nov 01 20:13:08.753623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.753553 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb8bbec525ef2fe80358c204eebb3a3c0bdd43ffad93552b5041392c9797a9bc" Nov 01 20:13:08.753623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.753604 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj" Nov 01 20:13:08.756191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:08.756166 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" event={"ID":"5696e679-5d84-4352-8122-42262f1457e0","Type":"ContainerStarted","Data":"4a0a88b9682093d135b969685a90c29360f3d4737ee9cf28feec15cc2c8f2c5e"} Nov 01 20:13:09.073978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.073794 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:13:09.073978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.073840 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:13:09.073978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.073975 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 20:13:09.114361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.113426 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:09.248358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.248120 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:09.264489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.264449 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:13:09.264665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.264643 5633 patch_prober.go:29] interesting pod/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 20:13:09.264719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.264688 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 20:13:09.904360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.900291 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw"] Nov 01 20:13:09.917554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.911302 5633 cpu_manager.go:414] "containerMap: removing container" podUID="180210f7-1850-414d-9c2e-5fb3f07f07ee" containerName="revoke-keys" Nov 01 20:13:09.917554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.911351 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180210f7-1850-414d-9c2e-5fb3f07f07ee" containerName="revoke-keys" Nov 01 20:13:09.917554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.911545 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="180210f7-1850-414d-9c2e-5fb3f07f07ee" containerName="revoke-keys" Nov 01 20:13:09.931360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.929565 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8"] Nov 01 20:13:09.931560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.931520 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:09.954466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.953962 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:09.955400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.955373 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7"] Nov 01 20:13:09.958211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:09.958189 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.027545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.027502 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw"] Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.066923 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.066981 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067014 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067043 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067075 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c113b0da-2c36-4f5d-8557-390ba1c244b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067101 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2f318-5931-41e9-93c2-612b264b8ed5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067133 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787wh\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-kube-api-access-787wh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067171 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067202 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067233 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067272 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067302 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.067354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067347 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkhr\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-kube-api-access-svkhr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067374 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067406 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067427 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067460 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067501 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067530 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067560 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067588 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrtt\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-kube-api-access-xbrtt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067620 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067646 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067698 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067729 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067778 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.068105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.067808 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.153374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.086739 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:13:10.153374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.086809 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 20:13:10.182549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.182450 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8"] Nov 01 20:13:10.184438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184400 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.184588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184516 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.184631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184590 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.184666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184635 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184731 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c113b0da-2c36-4f5d-8557-390ba1c244b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184811 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2f318-5931-41e9-93c2-612b264b8ed5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184852 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787wh\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-kube-api-access-787wh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184898 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184938 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.184981 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185003 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185036 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185077 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185110 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkhr\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-kube-api-access-svkhr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185147 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185195 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185229 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.188382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185268 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.239401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185319 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.278784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.278745 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.278966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185688 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.278966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.278814 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.278966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.278864 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.278966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.278892 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrtt\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-kube-api-access-xbrtt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.278966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.278928 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.278966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.278952 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.279284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279008 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.279284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279041 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.279284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279092 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.279284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279122 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.279513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279457 5633 patch_prober.go:29] interesting pod/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 20:13:10.279567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279507 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 20:13:10.279625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279579 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.279625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279592 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.279625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.187024 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.279625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185318 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.279826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.187179 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.280360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279886 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.280360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.186658 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.280360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.186754 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.280360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.187742 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.280360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.186416 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c113b0da-2c36-4f5d-8557-390ba1c244b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.280360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.280129 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.280360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.280172 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.280778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.278699 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.280778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.185794 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2f318-5931-41e9-93c2-612b264b8ed5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.284681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.281674 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.284681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.284232 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.285292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.279912 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.285292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.284968 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7"] Nov 01 20:13:10.300522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.285888 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.320392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.320070 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787wh\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-kube-api-access-787wh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.374101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.374047 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:10.386239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.386135 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.438200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.438101 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkhr\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-kube-api-access-svkhr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wd9hw\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.438434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.438401 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.442366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.438807 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lwr7\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:10.471798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.471376 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrtt\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-kube-api-access-xbrtt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rphz8\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.557709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.557667 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:10.573106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.573047 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:10.585095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:10.585033 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:11.109361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.083261 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:13:11.109361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.083357 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 20:13:11.277738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.275499 5633 patch_prober.go:29] interesting pod/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 20:13:11.277738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.275581 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 20:13:11.372379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.371577 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw"] Nov 01 20:13:11.374492 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:13:11.374457 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbfb2e8_b991_464c_8145_6a07f6250f6f.slice/crio-3f46eea56a084594d62e7bb6de3fc10a7f56c64f7f3914657ec5b01ee4bd9773 WatchSource:0}: Error finding container 3f46eea56a084594d62e7bb6de3fc10a7f56c64f7f3914657ec5b01ee4bd9773: Status 404 returned error can't find the container with id 3f46eea56a084594d62e7bb6de3fc10a7f56c64f7f3914657ec5b01ee4bd9773 Nov 01 20:13:11.378001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.377963 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.378261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.378218 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.378476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.378453 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.622149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.622045 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7"] Nov 01 20:13:11.636544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.635291 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.636544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.635404 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.636544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.635451 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.665744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.665308 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8"] Nov 01 20:13:11.673965 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:13:11.673928 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc113b0da_2c36_4f5d_8557_390ba1c244b5.slice/crio-3aeaca5a0492930d4d7fe5d61eb3941942b817dfc02813f7f473da4c98fd488b WatchSource:0}: Error finding container 3aeaca5a0492930d4d7fe5d61eb3941942b817dfc02813f7f473da4c98fd488b: Status 404 returned error can't find the container with id 3aeaca5a0492930d4d7fe5d61eb3941942b817dfc02813f7f473da4c98fd488b Nov 01 20:13:11.684911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.684298 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.684911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.684420 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.684911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.684467 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:11.853837 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.853777 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" event={"ID":"7fe2f318-5931-41e9-93c2-612b264b8ed5","Type":"ContainerStarted","Data":"5246963d83b51a5fd31979b7f3a5ffc1f925e4f1c62c63cf8e0b38cc61753fce"} Nov 01 20:13:11.865007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.864970 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" event={"ID":"c113b0da-2c36-4f5d-8557-390ba1c244b5","Type":"ContainerStarted","Data":"3aeaca5a0492930d4d7fe5d61eb3941942b817dfc02813f7f473da4c98fd488b"} Nov 01 20:13:11.865266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.865235 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" event={"ID":"5fbfb2e8-b991-464c-8145-6a07f6250f6f","Type":"ContainerStarted","Data":"3623057e348f7ba626b215c9786b0198a411f1b16c927e79ed1a687c8a9cb76f"} Nov 01 20:13:11.865631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:11.865600 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" event={"ID":"5fbfb2e8-b991-464c-8145-6a07f6250f6f","Type":"ContainerStarted","Data":"3f46eea56a084594d62e7bb6de3fc10a7f56c64f7f3914657ec5b01ee4bd9773"} Nov 01 20:13:12.110466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.106780 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:13:12.110466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.106878 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 20:13:12.191882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.191769 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:12.267387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.266747 5633 patch_prober.go:29] interesting pod/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 20:13:12.267387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.266813 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 20:13:12.642367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.581558 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:12.642367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.581620 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:12.642367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:12.600722 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:13.075893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.075853 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:13:13.076201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.076169 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 20:13:13.112294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.112247 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" event={"ID":"c113b0da-2c36-4f5d-8557-390ba1c244b5","Type":"ContainerStarted","Data":"21ba42efdfa18e508592940a8f92ea029ca415bad4b01986bfcb5e55310e9928"} Nov 01 20:13:13.206139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.206086 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" event={"ID":"7fe2f318-5931-41e9-93c2-612b264b8ed5","Type":"ContainerStarted","Data":"315dc25c30eca1646a5f9fca79f0928acee11e7519955e21c583457a67dfeebf"} Nov 01 20:13:13.286357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.265267 5633 patch_prober.go:29] interesting pod/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 20:13:13.286694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.286637 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 20:13:13.326429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.321417 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podStartSLOduration=4.321396068 podStartE2EDuration="4.321396068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:13.265713622 +0000 UTC m=+600.939972900" watchObservedRunningTime="2026-11-01 20:13:13.321396068 +0000 UTC m=+600.995655328" Nov 01 20:13:13.326429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.321631 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podStartSLOduration=4.32161663 podStartE2EDuration="4.32161663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:12.066803462 +0000 UTC m=+599.741062717" watchObservedRunningTime="2026-11-01 20:13:13.32161663 +0000 UTC m=+600.995875910" Nov 01 20:13:13.414127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.414064 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podStartSLOduration=4.41404273 podStartE2EDuration="4.41404273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:13.394901852 +0000 UTC m=+601.069161108" watchObservedRunningTime="2026-11-01 20:13:13.41404273 +0000 UTC m=+601.088301986" Nov 01 20:13:13.523751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.523714 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:13.558735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.558699 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:13.560191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.560153 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:13.584410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.579504 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:13.584410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.580803 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:13.584410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.583494 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:13.590008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.589976 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:13.590367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.590323 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:13.590542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:13.590519 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:14.074359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.074292 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 20:13:14.074602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.074386 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 20:13:14.153356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.153081 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:14.273357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.272772 5633 patch_prober.go:29] interesting pod/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 20:13:14.273357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.272836 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 20:13:14.297363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.294870 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:14.566371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.561783 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:14.566371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.561850 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:14.576206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.576085 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:14.576206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.576148 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:14.608372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.601746 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:14.608372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:14.601810 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:15.361856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.361805 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:13:15.364281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.364259 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:13:15.383377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.379646 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:13:15.560365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.560204 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:15.560365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.560265 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:15.580408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.577725 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:15.580408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.577788 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:15.586804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.586760 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:15.587063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:15.587036 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:16.090633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.090594 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:13:16.090940 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.090906 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:13:16.108793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.108748 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:13:16.110523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.110499 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:13:16.123675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.123637 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:13:16.356356 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:13:16.356227 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:13:16.563425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.558786 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:16.563425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.558858 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:16.574691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.574656 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:16.574878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.574705 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:16.651412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.603805 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:16.651412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.603897 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:16.840357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:16.839380 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:17.558786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:17.558744 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:17.559069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:17.559042 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:17.579781 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:17.579742 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:17.580385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:17.580319 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:17.586748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:17.586715 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:17.586955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:17.586931 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:17.607231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:17.601322 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:13:18.559066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:18.559026 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:18.559422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:18.559387 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:18.580801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:18.580766 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:18.581410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:18.581374 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:18.585807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:18.585770 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:18.586027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:18.585997 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:19.558765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:19.558724 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:19.559055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:19.559024 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:19.574373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:19.574317 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:19.574545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:19.574401 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:19.585653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:19.585617 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:19.586250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:19.586217 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:19.909257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:19.909102 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:20.559837 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.559795 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:20.560060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.559871 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:20.574303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.574260 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:20.574528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.574353 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:20.585546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.585503 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:20.585751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.585572 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:20.593981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.593944 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:20.951436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:20.951314 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:21.558807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.558760 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wd9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:13:21.559059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.558834 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 20:13:21.574290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.574248 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:21.574489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.574354 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:21.585777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.585736 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:13:21.586280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.585805 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 20:13:21.660577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.660524 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:21.867493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:21.867370 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:22.236992 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:22.236948 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:22.573935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:22.573826 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rphz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 20:13:22.573935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:22.573896 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 20:13:23.084069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.084034 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:23.086391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.086366 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:23.089069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.089049 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:13:23.628557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.628488 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lwr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:13:23.628777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.628578 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:13:23.855523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.855491 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:23.855964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.855940 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:23.856968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:23.856945 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:13:24.063585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:24.063552 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:24.064030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:24.064009 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:24.064901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:24.064874 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:13:25.887668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:25.887632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:26.224851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.224805 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:26.410773 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.410734 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:26.467128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.467097 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4"] Nov 01 20:13:26.469777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.469757 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.519360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.519294 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.519659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.519638 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.519828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.519814 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.519953 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.519939 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfp5\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-kube-api-access-9hfp5\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.520530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.520482 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.521118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.521088 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f1c85-71ca-4ca3-b151-c2407fb42882-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.521672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.521652 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.522047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.522027 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.522352 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.522317 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.522663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.521895 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-df41b8-openshift-default-dockercfg-75q9g\"" Nov 01 20:13:26.560792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.560759 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:26.574483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.574451 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4"] Nov 01 20:13:26.623440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623398 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623453 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623495 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623538 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623568 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfp5\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-kube-api-access-9hfp5\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623703 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623780 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f1c85-71ca-4ca3-b151-c2407fb42882-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623823 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.623994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.623888 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.624716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.624685 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.624976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.624954 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.625210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.625191 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.626251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.626200 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.628007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.627980 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f1c85-71ca-4ca3-b151-c2407fb42882-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.634165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.634090 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.637041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.637005 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.681270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.681229 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.689952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.689922 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfp5\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-kube-api-access-9hfp5\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.746909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.746819 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:26.784589 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.784558 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:26.886526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:26.886490 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:27.211409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.211380 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4"] Nov 01 20:13:27.250279 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:13:27.250247 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00f1c85_71ca_4ca3_b151_c2407fb42882.slice/crio-a6486b3503ef1d7f870dcde276b0c7b2993326a28f1f2bdb1df251cedbe599bc WatchSource:0}: Error finding container a6486b3503ef1d7f870dcde276b0c7b2993326a28f1f2bdb1df251cedbe599bc: Status 404 returned error can't find the container with id a6486b3503ef1d7f870dcde276b0c7b2993326a28f1f2bdb1df251cedbe599bc Nov 01 20:13:27.252868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.252835 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:27.252959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.252906 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:27.252959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.252936 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:13:27.275038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.275008 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:27.413664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.413621 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" event={"ID":"f00f1c85-71ca-4ca3-b151-c2407fb42882","Type":"ContainerStarted","Data":"61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8"} Nov 01 20:13:27.413825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.413691 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" event={"ID":"f00f1c85-71ca-4ca3-b151-c2407fb42882","Type":"ContainerStarted","Data":"a6486b3503ef1d7f870dcde276b0c7b2993326a28f1f2bdb1df251cedbe599bc"} Nov 01 20:13:27.530128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.530023 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podStartSLOduration=1.530007823 podStartE2EDuration="1.530007823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:27.528486114 +0000 UTC m=+615.202745369" watchObservedRunningTime="2026-11-01 20:13:27.530007823 +0000 UTC m=+615.204267139" Nov 01 20:13:27.785658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:27.785564 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:28.790755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:28.790724 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:29.395284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:29.395244 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:29.425874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:29.425833 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:29.426999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:29.426971 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:13:29.587366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:29.584875 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:30.958964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:30.958929 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:31.151844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:31.151805 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:32.620368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:32.620319 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:34.319312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.319276 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:34.493169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.493126 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:34.712195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.712164 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4"] Nov 01 20:13:34.716306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.716289 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:34.818806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.818771 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-df41b8-dockercfg-mt969\"" Nov 01 20:13:34.819034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.818787 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-df41b8\"" Nov 01 20:13:34.821814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.821787 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4"] Nov 01 20:13:34.822455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.822428 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29xd\" (UniqueName: \"kubernetes.io/projected/e346846b-8520-402b-87c2-84eb4d32e60f-kube-api-access-r29xd\") pod \"payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:34.822577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.822560 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e346846b-8520-402b-87c2-84eb4d32e60f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:34.923380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.923319 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e346846b-8520-402b-87c2-84eb4d32e60f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:34.923552 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.923500 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29xd\" (UniqueName: \"kubernetes.io/projected/e346846b-8520-402b-87c2-84eb4d32e60f-kube-api-access-r29xd\") pod \"payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:34.924032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.924009 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e346846b-8520-402b-87c2-84eb4d32e60f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:34.981812 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:34.981740 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29xd\" (UniqueName: \"kubernetes.io/projected/e346846b-8520-402b-87c2-84eb4d32e60f-kube-api-access-r29xd\") pod \"payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:35.030867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:35.030831 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:35.406922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:35.406897 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4"] Nov 01 20:13:35.408940 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:13:35.408912 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode346846b_8520_402b_87c2_84eb4d32e60f.slice/crio-8d789ac2fb915cbe3d597b3279c59c8702f4dfb134341b0ac9f99e7b04ffd13d WatchSource:0}: Error finding container 8d789ac2fb915cbe3d597b3279c59c8702f4dfb134341b0ac9f99e7b04ffd13d: Status 404 returned error can't find the container with id 8d789ac2fb915cbe3d597b3279c59c8702f4dfb134341b0ac9f99e7b04ffd13d Nov 01 20:13:35.464052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:35.464015 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" event={"ID":"e346846b-8520-402b-87c2-84eb4d32e60f","Type":"ContainerStarted","Data":"8d789ac2fb915cbe3d597b3279c59c8702f4dfb134341b0ac9f99e7b04ffd13d"} Nov 01 20:13:36.321787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:36.321747 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:36.478815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:36.478774 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" event={"ID":"e346846b-8520-402b-87c2-84eb4d32e60f","Type":"ContainerStarted","Data":"ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661"} Nov 01 20:13:36.479260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:36.478840 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:36.596536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:36.596411 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" podStartSLOduration=2.596397464 podStartE2EDuration="2.596397464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:36.594909069 +0000 UTC m=+624.269168324" watchObservedRunningTime="2026-11-01 20:13:36.596397464 +0000 UTC m=+624.270656748" Nov 01 20:13:36.915672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:36.915579 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:37.040375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:37.040321 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:37.484467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:37.484424 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:41.304030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:41.303993 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:13:42.293456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:42.293418 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:45.490044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:45.490004 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:46.158765 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:13:46.158725 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:13:46.941592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:46.941554 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:47.130256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:47.130215 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:47.488944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:47.488915 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:13:51.892972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:51.892876 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:56.541888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:56.541854 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:56.691401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:56.691359 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:56.819242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:56.819154 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:13:58.775385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:13:58.775349 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:14:01.504915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:01.504869 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:06.233134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:06.233090 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:11.026792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:11.026753 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:13.401353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:13.401301 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:14:15.128421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.128382 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:15.468612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.468566 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:14:15.468771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.468643 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:14:15.753905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.753868 5633 generic.go:356] "Generic (PLEG): container finished" podID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerID="61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8" exitCode=0 Nov 01 20:14:15.754152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.753941 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" event={"ID":"f00f1c85-71ca-4ca3-b151-c2407fb42882","Type":"ContainerDied","Data":"61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8"} Nov 01 20:14:15.754466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.754448 5633 scope.go:122] "RemoveContainer" containerID="61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8" Nov 01 20:14:15.755865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.755835 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:15.755957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.755907 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:15.755995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.755967 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:15.917288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:15.917187 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:16.158702 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:16.158605 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:14:16.762806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:16.762771 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" event={"ID":"f00f1c85-71ca-4ca3-b151-c2407fb42882","Type":"ContainerStarted","Data":"c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c"} Nov 01 20:14:16.805357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:16.793396 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:14:16.810547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:16.810508 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:14:16.810741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:16.810567 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 20:14:17.806353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:17.795737 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:14:17.806353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:17.795802 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 20:14:18.795525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:18.795480 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:14:18.795867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:18.795832 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 20:14:19.797371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:19.796774 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:14:19.797371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:19.796840 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 20:14:20.728087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:20.728047 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:20.809435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:20.786463 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:14:20.809435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:20.786527 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 20:14:20.937650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:20.937606 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:21.787443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:21.785762 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:14:21.787443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:21.785845 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 20:14:21.891484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:21.891443 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:22.789908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:22.789874 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:14:22.790262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:22.790238 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:14:22.791177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:22.791155 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:14:25.895760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:25.895726 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:30.603398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:30.603358 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:34.607709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:34.607671 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:14:35.578159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:35.578122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:36.162620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:36.162579 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:36.623256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:36.623219 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:37.109867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.109824 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh"] Nov 01 20:14:37.114193 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.114174 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.176353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.176308 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2cc24d-openshift-default-dockercfg-b4lxl\"" Nov 01 20:14:37.201091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201049 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201102 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h947b\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-kube-api-access-h947b\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201184 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201217 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201249 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cd787f-33ca-49ad-968b-d86b755b5826-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201365 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201414 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201523 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cd787f-33ca-49ad-968b-d86b755b5826-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.201597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.201549 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.209506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.209477 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:37.227779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.227749 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh"] Nov 01 20:14:37.302284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302245 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302299 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h947b\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-kube-api-access-h947b\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302345 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302368 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302399 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cd787f-33ca-49ad-968b-d86b755b5826-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302426 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302450 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302577 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cd787f-33ca-49ad-968b-d86b755b5826-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302617 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.302851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302824 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.303024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302936 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.303024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.302991 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.303106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.303050 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.303541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.303515 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cd787f-33ca-49ad-968b-d86b755b5826-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.305125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.305099 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cd787f-33ca-49ad-968b-d86b755b5826-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.305476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.305459 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.358911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.358878 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.359061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.358938 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h947b\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-kube-api-access-h947b\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.427103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.427017 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:37.625872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.625836 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:37.855489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.855459 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh"] Nov 01 20:14:37.857280 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:14:37.857249 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cd787f_33ca_49ad_968b_d86b755b5826.slice/crio-d0695788e851c68161d52c084cadb4f7b348e6769644a1c1bf33dfbc7bacbc19 WatchSource:0}: Error finding container d0695788e851c68161d52c084cadb4f7b348e6769644a1c1bf33dfbc7bacbc19: Status 404 returned error can't find the container with id d0695788e851c68161d52c084cadb4f7b348e6769644a1c1bf33dfbc7bacbc19 Nov 01 20:14:37.859810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.859777 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:37.859917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.859844 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:37.859917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.859871 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:37.943996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.943955 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" event={"ID":"41cd787f-33ca-49ad-968b-d86b755b5826","Type":"ContainerStarted","Data":"5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274"} Nov 01 20:14:37.944155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:37.944022 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" event={"ID":"41cd787f-33ca-49ad-968b-d86b755b5826","Type":"ContainerStarted","Data":"d0695788e851c68161d52c084cadb4f7b348e6769644a1c1bf33dfbc7bacbc19"} Nov 01 20:14:38.061751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:38.061699 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podStartSLOduration=1.061686517 podStartE2EDuration="1.061686517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:38.060404394 +0000 UTC m=+685.734663651" watchObservedRunningTime="2026-11-01 20:14:38.061686517 +0000 UTC m=+685.735945771" Nov 01 20:14:38.427500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:38.427464 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:38.432887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:38.432859 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:38.951410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:38.951369 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:38.952466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:38.952444 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:14:40.580094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:40.580047 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:40.829638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:40.829589 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:41.083529 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:41.083474 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:41.220486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:41.220430 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:44.036579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.036545 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:44.206578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.206542 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:44.620873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.620842 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn"] Nov 01 20:14:44.630932 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.630897 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.669653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.669622 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn"] Nov 01 20:14:44.697073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.697035 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lj6\" (UniqueName: \"kubernetes.io/projected/0fe286da-6465-4d55-b062-0af4426ebf20-kube-api-access-t9lj6\") pod \"payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.697252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.697096 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe286da-6465-4d55-b062-0af4426ebf20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.715676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.715647 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-2cc24d\"" Nov 01 20:14:44.715870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.715677 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-2cc24d-dockercfg-zwmf9\"" Nov 01 20:14:44.762855 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.762821 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl"] Nov 01 20:14:44.769730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.769705 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:44.798075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.798043 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a96f92-3084-4be6-b7a8-7f5f89c257e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:44.798256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.798088 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lj6\" (UniqueName: \"kubernetes.io/projected/0fe286da-6465-4d55-b062-0af4426ebf20-kube-api-access-t9lj6\") pod \"payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.798256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.798179 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe286da-6465-4d55-b062-0af4426ebf20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.798256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.798231 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mh9\" (UniqueName: \"kubernetes.io/projected/61a96f92-3084-4be6-b7a8-7f5f89c257e4-kube-api-access-g8mh9\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:44.798914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.798896 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe286da-6465-4d55-b062-0af4426ebf20-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.823177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.823136 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl"] Nov 01 20:14:44.855577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.855540 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lj6\" (UniqueName: \"kubernetes.io/projected/0fe286da-6465-4d55-b062-0af4426ebf20-kube-api-access-t9lj6\") pod \"payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.899238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.899147 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mh9\" (UniqueName: \"kubernetes.io/projected/61a96f92-3084-4be6-b7a8-7f5f89c257e4-kube-api-access-g8mh9\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:44.899438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.899268 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a96f92-3084-4be6-b7a8-7f5f89c257e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:44.899995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.899974 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a96f92-3084-4be6-b7a8-7f5f89c257e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:44.951545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.951511 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:44.956083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:44.956054 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mh9\" (UniqueName: \"kubernetes.io/projected/61a96f92-3084-4be6-b7a8-7f5f89c257e4-kube-api-access-g8mh9\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:45.086130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.086088 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:45.210287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.210252 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:45.353351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.353299 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn"] Nov 01 20:14:45.354915 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:14:45.354872 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe286da_6465_4d55_b062_0af4426ebf20.slice/crio-a686bfaa2a9744323a24972f5345416e0f504dfc35e470a7fb50a61ecdb431c2 WatchSource:0}: Error finding container a686bfaa2a9744323a24972f5345416e0f504dfc35e470a7fb50a61ecdb431c2: Status 404 returned error can't find the container with id a686bfaa2a9744323a24972f5345416e0f504dfc35e470a7fb50a61ecdb431c2 Nov 01 20:14:45.474716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.474692 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl"] Nov 01 20:14:45.477555 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:14:45.477532 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a96f92_3084_4be6_b7a8_7f5f89c257e4.slice/crio-301a7a7322726c60158fbb9d7b1485e7f32d2ed39bd47096a52b94bb8d3474de WatchSource:0}: Error finding container 301a7a7322726c60158fbb9d7b1485e7f32d2ed39bd47096a52b94bb8d3474de: Status 404 returned error can't find the container with id 301a7a7322726c60158fbb9d7b1485e7f32d2ed39bd47096a52b94bb8d3474de Nov 01 20:14:45.748454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.748421 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7364-4312e6"] Nov 01 20:14:45.752097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.752077 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:45.804558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.804527 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wlz8j\"" Nov 01 20:14:45.806571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.806541 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7364-4312e6"] Nov 01 20:14:45.812603 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.812557 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:45.813153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.813120 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvvw\" (UniqueName: \"kubernetes.io/projected/0283bc7a-f302-4e0a-a474-b67b15d9ec8e-kube-api-access-jhvvw\") pod \"test-curl-7364-4312e6\" (UID: \"0283bc7a-f302-4e0a-a474-b67b15d9ec8e\") " pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:45.914156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.914119 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvvw\" (UniqueName: \"kubernetes.io/projected/0283bc7a-f302-4e0a-a474-b67b15d9ec8e-kube-api-access-jhvvw\") pod \"test-curl-7364-4312e6\" (UID: \"0283bc7a-f302-4e0a-a474-b67b15d9ec8e\") " pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:45.970791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.970749 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvvw\" (UniqueName: \"kubernetes.io/projected/0283bc7a-f302-4e0a-a474-b67b15d9ec8e-kube-api-access-jhvvw\") pod \"test-curl-7364-4312e6\" (UID: \"0283bc7a-f302-4e0a-a474-b67b15d9ec8e\") " pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:45.999771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.999670 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" event={"ID":"61a96f92-3084-4be6-b7a8-7f5f89c257e4","Type":"ContainerStarted","Data":"6c999acebff5dd60b2fe244ecd08180c1e3f9974e939231ae089e9f10364c26b"} Nov 01 20:14:45.999771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.999737 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" event={"ID":"61a96f92-3084-4be6-b7a8-7f5f89c257e4","Type":"ContainerStarted","Data":"301a7a7322726c60158fbb9d7b1485e7f32d2ed39bd47096a52b94bb8d3474de"} Nov 01 20:14:45.999994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:45.999913 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:46.001526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.001499 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" event={"ID":"0fe286da-6465-4d55-b062-0af4426ebf20","Type":"ContainerStarted","Data":"10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1"} Nov 01 20:14:46.001643 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.001544 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" event={"ID":"0fe286da-6465-4d55-b062-0af4426ebf20","Type":"ContainerStarted","Data":"a686bfaa2a9744323a24972f5345416e0f504dfc35e470a7fb50a61ecdb431c2"} Nov 01 20:14:46.001643 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.001571 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:46.064539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.064497 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:46.125772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.125726 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" podStartSLOduration=2.125710406 podStartE2EDuration="2.125710406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:46.122360666 +0000 UTC m=+693.796619921" watchObservedRunningTime="2026-11-01 20:14:46.125710406 +0000 UTC m=+693.799969660" Nov 01 20:14:46.164999 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:46.164957 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:14:46.239484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.238743 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" podStartSLOduration=2.238721904 podStartE2EDuration="2.238721904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:46.238405742 +0000 UTC m=+693.912665001" watchObservedRunningTime="2026-11-01 20:14:46.238721904 +0000 UTC m=+693.912981160" Nov 01 20:14:46.436242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.436218 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7364-4312e6"] Nov 01 20:14:46.438018 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:14:46.437983 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0283bc7a_f302_4e0a_a474_b67b15d9ec8e.slice/crio-2681e1596ea6c6a79f7d6ce829524295418b4b2b7560b61cf7140b8cd11c1b1f WatchSource:0}: Error finding container 2681e1596ea6c6a79f7d6ce829524295418b4b2b7560b61cf7140b8cd11c1b1f: Status 404 returned error can't find the container with id 2681e1596ea6c6a79f7d6ce829524295418b4b2b7560b61cf7140b8cd11c1b1f Nov 01 20:14:46.440098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:46.440078 5633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:14:47.009234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:47.009200 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7364-4312e6" event={"ID":"0283bc7a-f302-4e0a-a474-b67b15d9ec8e","Type":"ContainerStarted","Data":"2681e1596ea6c6a79f7d6ce829524295418b4b2b7560b61cf7140b8cd11c1b1f"} Nov 01 20:14:48.017360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:48.017301 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7364-4312e6" event={"ID":"0283bc7a-f302-4e0a-a474-b67b15d9ec8e","Type":"ContainerStarted","Data":"e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0"} Nov 01 20:14:48.128905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:48.128848 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7364-4312e6" podStartSLOduration=2.016389598 podStartE2EDuration="3.128832672s" totalImagesPullingTime="1.112443074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:48.128272696 +0000 UTC m=+695.802531954" watchObservedRunningTime="2026-11-01 20:14:48.128832672 +0000 UTC m=+695.803091927" Nov 01 20:14:49.490051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.489907 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7364-4312e6"] Nov 01 20:14:49.494307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.494264 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7364-4312e6"] Nov 01 20:14:49.636017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.635983 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7364-23bcb4"] Nov 01 20:14:49.636583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.636569 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0283bc7a-f302-4e0a-a474-b67b15d9ec8e" containerName="test-curl-7364-4312e6" Nov 01 20:14:49.636638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.636585 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0283bc7a-f302-4e0a-a474-b67b15d9ec8e" containerName="test-curl-7364-4312e6" Nov 01 20:14:49.636697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.636687 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0283bc7a-f302-4e0a-a474-b67b15d9ec8e" containerName="test-curl-7364-4312e6" Nov 01 20:14:49.640147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.640131 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:49.694942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.694908 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7364-23bcb4"] Nov 01 20:14:49.758168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.758085 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblb7\" (UniqueName: \"kubernetes.io/projected/39136408-fb02-44a3-bbb6-0ce69abbf00f-kube-api-access-xblb7\") pod \"test-curl-7364-23bcb4\" (UID: \"39136408-fb02-44a3-bbb6-0ce69abbf00f\") " pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:49.859020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.858984 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xblb7\" (UniqueName: \"kubernetes.io/projected/39136408-fb02-44a3-bbb6-0ce69abbf00f-kube-api-access-xblb7\") pod \"test-curl-7364-23bcb4\" (UID: \"39136408-fb02-44a3-bbb6-0ce69abbf00f\") " pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:49.915652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.915614 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblb7\" (UniqueName: \"kubernetes.io/projected/39136408-fb02-44a3-bbb6-0ce69abbf00f-kube-api-access-xblb7\") pod \"test-curl-7364-23bcb4\" (UID: \"39136408-fb02-44a3-bbb6-0ce69abbf00f\") " pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:49.951866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:49.951833 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:50.031993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:50.031930 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7364-4312e6" podUID="0283bc7a-f302-4e0a-a474-b67b15d9ec8e" containerName="test-curl-7364-4312e6" containerID="cri-o://e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0" gracePeriod=2 Nov 01 20:14:50.082708 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:50.082667 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7364-4312e6\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="0283bc7a-f302-4e0a-a474-b67b15d9ec8e" pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:50.326520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:50.326487 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7364-23bcb4"] Nov 01 20:14:50.328115 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:14:50.328074 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39136408_fb02_44a3_bbb6_0ce69abbf00f.slice/crio-ac06128e4f3834c54860bfba3957d3960805cb071098da0fb8a9f1a29f1e8276 WatchSource:0}: Error finding container ac06128e4f3834c54860bfba3957d3960805cb071098da0fb8a9f1a29f1e8276: Status 404 returned error can't find the container with id ac06128e4f3834c54860bfba3957d3960805cb071098da0fb8a9f1a29f1e8276 Nov 01 20:14:50.513713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:50.513675 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:51.040730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:51.040685 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7364-23bcb4" event={"ID":"39136408-fb02-44a3-bbb6-0ce69abbf00f","Type":"ContainerStarted","Data":"81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae"} Nov 01 20:14:51.040932 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:51.040886 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7364-23bcb4" event={"ID":"39136408-fb02-44a3-bbb6-0ce69abbf00f","Type":"ContainerStarted","Data":"ac06128e4f3834c54860bfba3957d3960805cb071098da0fb8a9f1a29f1e8276"} Nov 01 20:14:51.152504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:51.152449 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7364-23bcb4" podStartSLOduration=2.152435156 podStartE2EDuration="2.152435156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:51.151366632 +0000 UTC m=+698.825625885" watchObservedRunningTime="2026-11-01 20:14:51.152435156 +0000 UTC m=+698.826694410" Nov 01 20:14:52.243910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:52.243876 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:52.386767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:52.386674 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0283bc7a-f302-4e0a-a474-b67b15d9ec8e-kube-api-access-jhvvw\" (UniqueName: \"kubernetes.io/projected/0283bc7a-f302-4e0a-a474-b67b15d9ec8e-kube-api-access-jhvvw\") pod \"0283bc7a-f302-4e0a-a474-b67b15d9ec8e\" (UID: \"0283bc7a-f302-4e0a-a474-b67b15d9ec8e\") " Nov 01 20:14:52.389270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:52.389239 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0283bc7a-f302-4e0a-a474-b67b15d9ec8e-kube-api-access-jhvvw" pod "0283bc7a-f302-4e0a-a474-b67b15d9ec8e" (UID: "0283bc7a-f302-4e0a-a474-b67b15d9ec8e"). InnerVolumeSpecName "kube-api-access-jhvvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:14:52.467418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:52.467385 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0283bc7a-f302-4e0a-a474-b67b15d9ec8e" path="/var/lib/kubelet/pods/0283bc7a-f302-4e0a-a474-b67b15d9ec8e/volumes" Nov 01 20:14:52.487804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:52.487768 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvvw\" (UniqueName: \"kubernetes.io/projected/0283bc7a-f302-4e0a-a474-b67b15d9ec8e-kube-api-access-jhvvw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:14:52.624037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:52.624003 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7364-23bcb4"] Nov 01 20:14:52.631616 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:52.631578 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7364-23bcb4"] Nov 01 20:14:53.055319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.055282 5633 generic.go:356] "Generic (PLEG): container finished" podID="0283bc7a-f302-4e0a-a474-b67b15d9ec8e" containerID="e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0" exitCode=137 Nov 01 20:14:53.055564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.055358 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-4312e6" Nov 01 20:14:53.055564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.055399 5633 scope.go:122] "RemoveContainer" containerID="e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0" Nov 01 20:14:53.055691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.055628 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7364-23bcb4" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" containerName="test-curl-7364-23bcb4" containerID="cri-o://81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae" gracePeriod=2 Nov 01 20:14:53.065301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.065271 5633 scope.go:122] "RemoveContainer" containerID="e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0" Nov 01 20:14:53.065663 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:53.065642 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0\": container with ID starting with e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0 not found: ID does not exist" containerID="e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0" Nov 01 20:14:53.065723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.065675 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0"} err="failed to get container status \"e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0\": rpc error: code = NotFound desc = could not find container \"e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0\": container with ID starting with e22a6ceeb3415271be3fd7fe76865af34412e4337200f18d4f0cd76c9242a7e0 not found: ID does not exist" Nov 01 20:14:53.106494 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:53.106449 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7364-23bcb4\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:53.157144 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:53.157102 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7364-23bcb4\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:53.345024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.344913 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:53.854989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.854949 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv"] Nov 01 20:14:53.855652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.855628 5633 cpu_manager.go:414] "containerMap: removing container" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" containerName="test-curl-7364-23bcb4" Nov 01 20:14:53.855652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.855650 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" containerName="test-curl-7364-23bcb4" Nov 01 20:14:53.855853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.855775 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" containerName="test-curl-7364-23bcb4" Nov 01 20:14:53.860403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.860376 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:53.916586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.916536 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:14:53.942048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.942006 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:53.988297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:53.988253 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv"] Nov 01 20:14:54.000656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.000627 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ad7edb-openshift-default-dockercfg-hkt6v\"" Nov 01 20:14:54.009014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.008969 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-podinfo\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.009318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.009298 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-certs\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.009582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.009564 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.009738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.009723 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113e6f55-ced3-4c66-b51d-83f8bc007233-istiod-ca-cert\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.009858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.009842 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-credential-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.010033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.010014 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-data\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.010159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.010142 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwsz\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-kube-api-access-5dwsz\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.010294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.010281 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-envoy\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.010437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.010424 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-token\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.068357 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:54.039783 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7364-23bcb4\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:54.133413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133062 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113e6f55-ced3-4c66-b51d-83f8bc007233-istiod-ca-cert\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133125 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-credential-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133190 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-data\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133227 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwsz\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-kube-api-access-5dwsz\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133277 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-envoy\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133347 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-token\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133406 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-podinfo\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133463 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-certs\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.133893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133508 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.134065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.133979 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.134692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.134668 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113e6f55-ced3-4c66-b51d-83f8bc007233-istiod-ca-cert\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.135099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.135079 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-credential-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.135471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.135453 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-data\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.136820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.136775 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-certs\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.140301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.140233 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:54.174820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.174786 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-envoy\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.175987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.175963 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-podinfo\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.210359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.201267 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwsz\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-kube-api-access-5dwsz\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.218563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.218500 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-token\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.473496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.473466 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:54.984101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.984070 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv"] Nov 01 20:14:54.996248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.995971 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:54.996248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.996069 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:54.996248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:54.996112 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:14:55.123942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:55.123765 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" event={"ID":"113e6f55-ced3-4c66-b51d-83f8bc007233","Type":"ContainerStarted","Data":"1f50d8176c0204ddc161d887975240f4411800bfd47bd32e6c6e6cc22d63a5c7"} Nov 01 20:14:55.363747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:55.363715 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:55.395356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:55.394655 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:55.467072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:55.467034 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39136408-fb02-44a3-bbb6-0ce69abbf00f-kube-api-access-xblb7\" (UniqueName: \"kubernetes.io/projected/39136408-fb02-44a3-bbb6-0ce69abbf00f-kube-api-access-xblb7\") pod \"39136408-fb02-44a3-bbb6-0ce69abbf00f\" (UID: \"39136408-fb02-44a3-bbb6-0ce69abbf00f\") " Nov 01 20:14:55.473267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:55.473226 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39136408-fb02-44a3-bbb6-0ce69abbf00f-kube-api-access-xblb7" pod "39136408-fb02-44a3-bbb6-0ce69abbf00f" (UID: "39136408-fb02-44a3-bbb6-0ce69abbf00f"). InnerVolumeSpecName "kube-api-access-xblb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:14:55.569623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:55.569537 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xblb7\" (UniqueName: \"kubernetes.io/projected/39136408-fb02-44a3-bbb6-0ce69abbf00f-kube-api-access-xblb7\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:14:56.132438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.132400 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" event={"ID":"113e6f55-ced3-4c66-b51d-83f8bc007233","Type":"ContainerStarted","Data":"bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569"} Nov 01 20:14:56.133920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.133893 5633 generic.go:356] "Generic (PLEG): container finished" podID="39136408-fb02-44a3-bbb6-0ce69abbf00f" containerID="81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae" exitCode=137 Nov 01 20:14:56.134070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.133971 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:56.134070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.133981 5633 scope.go:122] "RemoveContainer" containerID="81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae" Nov 01 20:14:56.148659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.148635 5633 scope.go:122] "RemoveContainer" containerID="81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae" Nov 01 20:14:56.149006 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:56.148981 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae\": container with ID starting with 81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae not found: ID does not exist" containerID="81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae" Nov 01 20:14:56.149074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.149021 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae"} err="failed to get container status \"81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae\": rpc error: code = NotFound desc = could not find container \"81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae\": container with ID starting with 81ef437cfcfdd344732de55e0bd1f690739d3b5d649e5fd23dbd8e8f07e2afae not found: ID does not exist" Nov 01 20:14:56.398322 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:14:56.398232 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7364-23bcb4\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" pod="opendatahub/test-curl-7364-23bcb4" Nov 01 20:14:56.467210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.467182 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39136408-fb02-44a3-bbb6-0ce69abbf00f" path="/var/lib/kubelet/pods/39136408-fb02-44a3-bbb6-0ce69abbf00f/volumes" Nov 01 20:14:56.474029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.474005 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:56.479432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.479400 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:56.601162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:56.601103 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" podStartSLOduration=3.601083227 podStartE2EDuration="3.601083227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:56.347442062 +0000 UTC m=+704.021701317" watchObservedRunningTime="2026-11-01 20:14:56.601083227 +0000 UTC m=+704.275342482" Nov 01 20:14:57.013153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:57.013118 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:14:57.013404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:57.013194 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:14:57.146280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:57.146245 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:57.147593 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:57.147566 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:14:59.079806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:59.079768 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:59.250390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:59.250309 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:14:59.399986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:14:59.399902 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:00.174082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.174030 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz"] Nov 01 20:15:00.179750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.179725 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx"] Nov 01 20:15:00.179900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.179847 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.183521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.183497 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9"] Nov 01 20:15:00.183664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.183620 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.187234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.187211 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.222805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.222751 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dgj2\" (UniqueName: \"kubernetes.io/projected/4d241298-cea0-40fc-87c0-6fddbbbe76f3-kube-api-access-8dgj2\") pod \"maas-api-key-cleanup-29892735-msdjz\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.222952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.222803 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4pb\" (UniqueName: \"kubernetes.io/projected/170adf98-24f7-4d09-b0fc-6e31e366b8e1-kube-api-access-vt4pb\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.222952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.222927 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d241298-cea0-40fc-87c0-6fddbbbe76f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892735-msdjz\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.223042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.222983 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170adf98-24f7-4d09-b0fc-6e31e366b8e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.223077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.223049 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b07853c-178d-4f4b-92f4-3040eaa79f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.223174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.223154 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw98k\" (UniqueName: \"kubernetes.io/projected/4b07853c-178d-4f4b-92f4-3040eaa79f90-kube-api-access-qw98k\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.223535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.223519 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl"] Nov 01 20:15:00.227456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.227437 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.237217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.237186 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz"] Nov 01 20:15:00.239269 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.239241 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9"] Nov 01 20:15:00.244013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.243984 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx"] Nov 01 20:15:00.270469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.270438 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:15:00.270624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.270585 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:15:00.281355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.281306 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:15:00.284122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.284085 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl"] Nov 01 20:15:00.324286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324244 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170adf98-24f7-4d09-b0fc-6e31e366b8e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.324548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324298 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b07853c-178d-4f4b-92f4-3040eaa79f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.324548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324405 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw98k\" (UniqueName: \"kubernetes.io/projected/4b07853c-178d-4f4b-92f4-3040eaa79f90-kube-api-access-qw98k\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.324548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324458 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dgj2\" (UniqueName: \"kubernetes.io/projected/4d241298-cea0-40fc-87c0-6fddbbbe76f3-kube-api-access-8dgj2\") pod \"maas-api-key-cleanup-29892735-msdjz\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.324548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324484 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4pb\" (UniqueName: \"kubernetes.io/projected/170adf98-24f7-4d09-b0fc-6e31e366b8e1-kube-api-access-vt4pb\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.324740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324608 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003def20-e339-4b32-a68c-1b28e7980024-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.324740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324644 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d241298-cea0-40fc-87c0-6fddbbbe76f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892735-msdjz\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.324740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.324672 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4wf\" (UniqueName: \"kubernetes.io/projected/003def20-e339-4b32-a68c-1b28e7980024-kube-api-access-6h4wf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.325087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.325062 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170adf98-24f7-4d09-b0fc-6e31e366b8e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.325194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.325135 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b07853c-178d-4f4b-92f4-3040eaa79f90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.325366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.325322 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d241298-cea0-40fc-87c0-6fddbbbe76f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892735-msdjz\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.390990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.382569 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4pb\" (UniqueName: \"kubernetes.io/projected/170adf98-24f7-4d09-b0fc-6e31e366b8e1-kube-api-access-vt4pb\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.390990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.387513 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw98k\" (UniqueName: \"kubernetes.io/projected/4b07853c-178d-4f4b-92f4-3040eaa79f90-kube-api-access-qw98k\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.390990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.390896 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dgj2\" (UniqueName: \"kubernetes.io/projected/4d241298-cea0-40fc-87c0-6fddbbbe76f3-kube-api-access-8dgj2\") pod \"maas-api-key-cleanup-29892735-msdjz\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.418945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.418906 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:00.426289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.426214 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003def20-e339-4b32-a68c-1b28e7980024-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.426289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.426265 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4wf\" (UniqueName: \"kubernetes.io/projected/003def20-e339-4b32-a68c-1b28e7980024-kube-api-access-6h4wf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.426865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.426845 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003def20-e339-4b32-a68c-1b28e7980024-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.484538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.484493 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4wf\" (UniqueName: \"kubernetes.io/projected/003def20-e339-4b32-a68c-1b28e7980024-kube-api-access-6h4wf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.495155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.495114 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:00.503652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.503615 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:00.511247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.511216 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:00.549233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.549134 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:00.878612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.878579 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz"] Nov 01 20:15:00.879893 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:00.879868 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d241298_cea0_40fc_87c0_6fddbbbe76f3.slice/crio-5a7b0f8c73f40c8f488c5c124d9049a4d613581e3951bf3dbb07e63e42736f50 WatchSource:0}: Error finding container 5a7b0f8c73f40c8f488c5c124d9049a4d613581e3951bf3dbb07e63e42736f50: Status 404 returned error can't find the container with id 5a7b0f8c73f40c8f488c5c124d9049a4d613581e3951bf3dbb07e63e42736f50 Nov 01 20:15:00.927753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.927722 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx"] Nov 01 20:15:00.929188 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:00.929152 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170adf98_24f7_4d09_b0fc_6e31e366b8e1.slice/crio-70a1d8efc1795aa12df9a38054eecb7cff16c930aa3aae96319164d3d25c5312 WatchSource:0}: Error finding container 70a1d8efc1795aa12df9a38054eecb7cff16c930aa3aae96319164d3d25c5312: Status 404 returned error can't find the container with id 70a1d8efc1795aa12df9a38054eecb7cff16c930aa3aae96319164d3d25c5312 Nov 01 20:15:00.930667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.930643 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9"] Nov 01 20:15:00.931081 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:00.931055 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b07853c_178d_4f4b_92f4_3040eaa79f90.slice/crio-eee35d4ebba63bba4738df362ee7f561e3ca4d525684507631dfa3649d3d9689 WatchSource:0}: Error finding container eee35d4ebba63bba4738df362ee7f561e3ca4d525684507631dfa3649d3d9689: Status 404 returned error can't find the container with id eee35d4ebba63bba4738df362ee7f561e3ca4d525684507631dfa3649d3d9689 Nov 01 20:15:00.979994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:00.979967 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl"] Nov 01 20:15:00.981475 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:00.981450 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003def20_e339_4b32_a68c_1b28e7980024.slice/crio-9cb66b1c2502e3929cfcd3248451a0ebf8808c913ccb5a22d5930a747c08d91d WatchSource:0}: Error finding container 9cb66b1c2502e3929cfcd3248451a0ebf8808c913ccb5a22d5930a747c08d91d: Status 404 returned error can't find the container with id 9cb66b1c2502e3929cfcd3248451a0ebf8808c913ccb5a22d5930a747c08d91d Nov 01 20:15:01.170964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:01.170928 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" event={"ID":"170adf98-24f7-4d09-b0fc-6e31e366b8e1","Type":"ContainerStarted","Data":"70a1d8efc1795aa12df9a38054eecb7cff16c930aa3aae96319164d3d25c5312"} Nov 01 20:15:01.172381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:01.172355 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" event={"ID":"003def20-e339-4b32-a68c-1b28e7980024","Type":"ContainerStarted","Data":"9cb66b1c2502e3929cfcd3248451a0ebf8808c913ccb5a22d5930a747c08d91d"} Nov 01 20:15:01.173695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:01.173654 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" event={"ID":"4d241298-cea0-40fc-87c0-6fddbbbe76f3","Type":"ContainerStarted","Data":"5a7b0f8c73f40c8f488c5c124d9049a4d613581e3951bf3dbb07e63e42736f50"} Nov 01 20:15:01.174957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:01.174932 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" event={"ID":"4b07853c-178d-4f4b-92f4-3040eaa79f90","Type":"ContainerStarted","Data":"eee35d4ebba63bba4738df362ee7f561e3ca4d525684507631dfa3649d3d9689"} Nov 01 20:15:02.187222 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.187177 5633 generic.go:356] "Generic (PLEG): container finished" podID="003def20-e339-4b32-a68c-1b28e7980024" containerID="de99684b7d665854579ee2c8a7a055968c4a0a1f6380ee413d7d7d201ca6ac42" exitCode=60 Nov 01 20:15:02.187717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.187272 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" event={"ID":"003def20-e339-4b32-a68c-1b28e7980024","Type":"ContainerDied","Data":"de99684b7d665854579ee2c8a7a055968c4a0a1f6380ee413d7d7d201ca6ac42"} Nov 01 20:15:02.187717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.187651 5633 scope.go:122] "RemoveContainer" containerID="de99684b7d665854579ee2c8a7a055968c4a0a1f6380ee413d7d7d201ca6ac42" Nov 01 20:15:02.189146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.189109 5633 generic.go:356] "Generic (PLEG): container finished" podID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerID="dd055fb8be4544d99294e3b830d3789b76e8d0ed6d2eda794a4fb6cf70de6371" exitCode=60 Nov 01 20:15:02.189258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.189178 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" event={"ID":"4d241298-cea0-40fc-87c0-6fddbbbe76f3","Type":"ContainerDied","Data":"dd055fb8be4544d99294e3b830d3789b76e8d0ed6d2eda794a4fb6cf70de6371"} Nov 01 20:15:02.189492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.189453 5633 scope.go:122] "RemoveContainer" containerID="dd055fb8be4544d99294e3b830d3789b76e8d0ed6d2eda794a4fb6cf70de6371" Nov 01 20:15:02.191051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.190975 5633 generic.go:356] "Generic (PLEG): container finished" podID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerID="8d7ccb74068d77fc73de0837f698bf70e2524d25749ae611043016e36393452a" exitCode=60 Nov 01 20:15:02.191111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.191083 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" event={"ID":"4b07853c-178d-4f4b-92f4-3040eaa79f90","Type":"ContainerDied","Data":"8d7ccb74068d77fc73de0837f698bf70e2524d25749ae611043016e36393452a"} Nov 01 20:15:02.191286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.191272 5633 scope.go:122] "RemoveContainer" containerID="8d7ccb74068d77fc73de0837f698bf70e2524d25749ae611043016e36393452a" Nov 01 20:15:02.193493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.193470 5633 generic.go:356] "Generic (PLEG): container finished" podID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerID="7898f9fa825f4e5ca2071db68d822df73e465fbca3918c17f1ccd5dec6cdb5b1" exitCode=60 Nov 01 20:15:02.193575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.193555 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" event={"ID":"170adf98-24f7-4d09-b0fc-6e31e366b8e1","Type":"ContainerDied","Data":"7898f9fa825f4e5ca2071db68d822df73e465fbca3918c17f1ccd5dec6cdb5b1"} Nov 01 20:15:02.193721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.193708 5633 scope.go:122] "RemoveContainer" containerID="7898f9fa825f4e5ca2071db68d822df73e465fbca3918c17f1ccd5dec6cdb5b1" Nov 01 20:15:02.307213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.307169 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:02.492121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.492082 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:02.759622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.759545 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2"] Nov 01 20:15:02.763615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.763597 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:02.817053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.817022 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ad7edb-dockercfg-jfvc8\"" Nov 01 20:15:02.817053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.817028 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ad7edb\"" Nov 01 20:15:02.820365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.820293 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2"] Nov 01 20:15:02.858976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.858852 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2jk\" (UniqueName: \"kubernetes.io/projected/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-kube-api-access-jj2jk\") pod \"payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:02.858976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.858953 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:02.960453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.960419 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2jk\" (UniqueName: \"kubernetes.io/projected/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-kube-api-access-jj2jk\") pod \"payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:02.960668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.960479 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:02.961184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:02.961160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:03.016800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.016726 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2jk\" (UniqueName: \"kubernetes.io/projected/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-kube-api-access-jj2jk\") pod \"payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:03.077326 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.077281 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:03.202504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.202458 5633 generic.go:356] "Generic (PLEG): container finished" podID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerID="f841dad1e2e01e552abce7c6565937d2548d8aec8baf3ba8afddabd31a555604" exitCode=60 Nov 01 20:15:03.202942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.202535 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" event={"ID":"4d241298-cea0-40fc-87c0-6fddbbbe76f3","Type":"ContainerDied","Data":"f841dad1e2e01e552abce7c6565937d2548d8aec8baf3ba8afddabd31a555604"} Nov 01 20:15:03.202942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.202596 5633 scope.go:122] "RemoveContainer" containerID="dd055fb8be4544d99294e3b830d3789b76e8d0ed6d2eda794a4fb6cf70de6371" Nov 01 20:15:03.202942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.202883 5633 scope.go:122] "RemoveContainer" containerID="f841dad1e2e01e552abce7c6565937d2548d8aec8baf3ba8afddabd31a555604" Nov 01 20:15:03.203439 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:03.203220 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892735-msdjz_odh-ai-gateway-infra(4d241298-cea0-40fc-87c0-6fddbbbe76f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" Nov 01 20:15:03.205948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.205365 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" event={"ID":"4b07853c-178d-4f4b-92f4-3040eaa79f90","Type":"ContainerStarted","Data":"6e7bcbe2e5bb70eeb2f2d08fde613f8c6c732c82d277e7c3637662a697cad022"} Nov 01 20:15:03.208325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.208294 5633 generic.go:356] "Generic (PLEG): container finished" podID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerID="ca5a1a44c42d514ab2337f4b160c545b2b0603f9d0f7e53f64e6041cbf0bc60a" exitCode=60 Nov 01 20:15:03.208459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.208442 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" event={"ID":"170adf98-24f7-4d09-b0fc-6e31e366b8e1","Type":"ContainerDied","Data":"ca5a1a44c42d514ab2337f4b160c545b2b0603f9d0f7e53f64e6041cbf0bc60a"} Nov 01 20:15:03.208774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.208756 5633 scope.go:122] "RemoveContainer" containerID="ca5a1a44c42d514ab2337f4b160c545b2b0603f9d0f7e53f64e6041cbf0bc60a" Nov 01 20:15:03.209066 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:03.209049 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx_odh-ai-gateway-infra(170adf98-24f7-4d09-b0fc-6e31e366b8e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" Nov 01 20:15:03.210743 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.210718 5633 generic.go:356] "Generic (PLEG): container finished" podID="003def20-e339-4b32-a68c-1b28e7980024" containerID="e32489f760b35a7418ead5dec8ce42241249323d7756462681a6f270e5b878cd" exitCode=60 Nov 01 20:15:03.210869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.210769 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" event={"ID":"003def20-e339-4b32-a68c-1b28e7980024","Type":"ContainerDied","Data":"e32489f760b35a7418ead5dec8ce42241249323d7756462681a6f270e5b878cd"} Nov 01 20:15:03.211055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.211041 5633 scope.go:122] "RemoveContainer" containerID="e32489f760b35a7418ead5dec8ce42241249323d7756462681a6f270e5b878cd" Nov 01 20:15:03.211325 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:03.211304 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl_odh-ai-gateway-infra(003def20-e339-4b32-a68c-1b28e7980024)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" podUID="003def20-e339-4b32-a68c-1b28e7980024" Nov 01 20:15:03.275767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.275743 5633 scope.go:122] "RemoveContainer" containerID="7898f9fa825f4e5ca2071db68d822df73e465fbca3918c17f1ccd5dec6cdb5b1" Nov 01 20:15:03.293229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.293203 5633 scope.go:122] "RemoveContainer" containerID="de99684b7d665854579ee2c8a7a055968c4a0a1f6380ee413d7d7d201ca6ac42" Nov 01 20:15:03.429704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.429649 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" podStartSLOduration=3.4296333949999998 podStartE2EDuration="3.429633395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:03.427189483 +0000 UTC m=+711.101448731" watchObservedRunningTime="2026-11-01 20:15:03.429633395 +0000 UTC m=+711.103892650" Nov 01 20:15:03.459028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.458995 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2"] Nov 01 20:15:03.460788 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:03.460758 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0baf5d_cbf6_444e_ab7a_972d89d1a3ae.slice/crio-91f119e72726fda9b884d5b5efa9b62cdce3456576c35bc8e60b228affd7ace0 WatchSource:0}: Error finding container 91f119e72726fda9b884d5b5efa9b62cdce3456576c35bc8e60b228affd7ace0: Status 404 returned error can't find the container with id 91f119e72726fda9b884d5b5efa9b62cdce3456576c35bc8e60b228affd7ace0 Nov 01 20:15:03.559001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:03.558969 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:04.219969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.219937 5633 generic.go:356] "Generic (PLEG): container finished" podID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerID="6e7bcbe2e5bb70eeb2f2d08fde613f8c6c732c82d277e7c3637662a697cad022" exitCode=60 Nov 01 20:15:04.220455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.220027 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" event={"ID":"4b07853c-178d-4f4b-92f4-3040eaa79f90","Type":"ContainerDied","Data":"6e7bcbe2e5bb70eeb2f2d08fde613f8c6c732c82d277e7c3637662a697cad022"} Nov 01 20:15:04.220455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.220091 5633 scope.go:122] "RemoveContainer" containerID="8d7ccb74068d77fc73de0837f698bf70e2524d25749ae611043016e36393452a" Nov 01 20:15:04.220455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.220268 5633 scope.go:122] "RemoveContainer" containerID="6e7bcbe2e5bb70eeb2f2d08fde613f8c6c732c82d277e7c3637662a697cad022" Nov 01 20:15:04.220626 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:04.220602 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9_odh-ai-gateway-infra(4b07853c-178d-4f4b-92f4-3040eaa79f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" Nov 01 20:15:04.222540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.222520 5633 scope.go:122] "RemoveContainer" containerID="ca5a1a44c42d514ab2337f4b160c545b2b0603f9d0f7e53f64e6041cbf0bc60a" Nov 01 20:15:04.222833 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:04.222809 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx_odh-ai-gateway-infra(170adf98-24f7-4d09-b0fc-6e31e366b8e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" Nov 01 20:15:04.223984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.223954 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" event={"ID":"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae","Type":"ContainerStarted","Data":"968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c"} Nov 01 20:15:04.224066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.223998 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" event={"ID":"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae","Type":"ContainerStarted","Data":"91f119e72726fda9b884d5b5efa9b62cdce3456576c35bc8e60b228affd7ace0"} Nov 01 20:15:04.224066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.224044 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:04.225914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.225892 5633 scope.go:122] "RemoveContainer" containerID="e32489f760b35a7418ead5dec8ce42241249323d7756462681a6f270e5b878cd" Nov 01 20:15:04.226227 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:04.226202 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl_odh-ai-gateway-infra(003def20-e339-4b32-a68c-1b28e7980024)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" podUID="003def20-e339-4b32-a68c-1b28e7980024" Nov 01 20:15:04.227428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.227411 5633 scope.go:122] "RemoveContainer" containerID="f841dad1e2e01e552abce7c6565937d2548d8aec8baf3ba8afddabd31a555604" Nov 01 20:15:04.227654 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:04.227627 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892735-msdjz_odh-ai-gateway-infra(4d241298-cea0-40fc-87c0-6fddbbbe76f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" Nov 01 20:15:04.791884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:04.791824 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" podStartSLOduration=2.7918088770000002 podStartE2EDuration="2.791808877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:04.789788791 +0000 UTC m=+712.464048047" watchObservedRunningTime="2026-11-01 20:15:04.791808877 +0000 UTC m=+712.466068184" Nov 01 20:15:05.235535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:05.235492 5633 scope.go:122] "RemoveContainer" containerID="6e7bcbe2e5bb70eeb2f2d08fde613f8c6c732c82d277e7c3637662a697cad022" Nov 01 20:15:05.235982 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:05.235850 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9_odh-ai-gateway-infra(4b07853c-178d-4f4b-92f4-3040eaa79f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" Nov 01 20:15:05.584880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:05.584777 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:10.233747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:10.233712 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:10.414493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:10.414442 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:10.541079 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:10.540993 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:12.398690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:12.398653 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:15:14.460602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:14.460569 5633 scope.go:122] "RemoveContainer" containerID="6e7bcbe2e5bb70eeb2f2d08fde613f8c6c732c82d277e7c3637662a697cad022" Nov 01 20:15:14.461078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:14.460633 5633 scope.go:122] "RemoveContainer" containerID="ca5a1a44c42d514ab2337f4b160c545b2b0603f9d0f7e53f64e6041cbf0bc60a" Nov 01 20:15:15.205704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.205608 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:15.238722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.238692 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:15:15.325123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.325081 5633 generic.go:356] "Generic (PLEG): container finished" podID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerID="beac0881f522440b7dcf440a8070cd122cac1353d08edab418d63e26339dbb95" exitCode=60 Nov 01 20:15:15.325281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.325156 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" event={"ID":"4b07853c-178d-4f4b-92f4-3040eaa79f90","Type":"ContainerDied","Data":"beac0881f522440b7dcf440a8070cd122cac1353d08edab418d63e26339dbb95"} Nov 01 20:15:15.325281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.325228 5633 scope.go:122] "RemoveContainer" containerID="6e7bcbe2e5bb70eeb2f2d08fde613f8c6c732c82d277e7c3637662a697cad022" Nov 01 20:15:15.325646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.325626 5633 scope.go:122] "RemoveContainer" containerID="beac0881f522440b7dcf440a8070cd122cac1353d08edab418d63e26339dbb95" Nov 01 20:15:15.325944 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:15.325924 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9_odh-ai-gateway-infra(4b07853c-178d-4f4b-92f4-3040eaa79f90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" Nov 01 20:15:15.327608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.327586 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" event={"ID":"170adf98-24f7-4d09-b0fc-6e31e366b8e1","Type":"ContainerStarted","Data":"8b04491a2395287e57cc2cd30c9d9497a85f2b1d3ec6b18255a2ae9f424ef3ef"} Nov 01 20:15:15.459687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.459617 5633 scope.go:122] "RemoveContainer" containerID="e32489f760b35a7418ead5dec8ce42241249323d7756462681a6f270e5b878cd" Nov 01 20:15:15.459820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.459710 5633 scope.go:122] "RemoveContainer" containerID="f841dad1e2e01e552abce7c6565937d2548d8aec8baf3ba8afddabd31a555604" Nov 01 20:15:15.579530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:15.579477 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" podStartSLOduration=15.579462445 podStartE2EDuration="15.579462445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:15.578474163 +0000 UTC m=+723.252733443" watchObservedRunningTime="2026-11-01 20:15:15.579462445 +0000 UTC m=+723.253721693" Nov 01 20:15:16.156829 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:16.156736 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:15:16.336672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.336627 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" event={"ID":"003def20-e339-4b32-a68c-1b28e7980024","Type":"ContainerStarted","Data":"6a659b5bcaf150d14762e4607a30552745bf7cec97d2c0098f5fc5f06026ac88"} Nov 01 20:15:16.338689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.338660 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" event={"ID":"4d241298-cea0-40fc-87c0-6fddbbbe76f3","Type":"ContainerStarted","Data":"055581f816db1a90dab4b14765f06b9ca28ecc59c98ecf718856774f37ee8243"} Nov 01 20:15:16.342081 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.342047 5633 generic.go:356] "Generic (PLEG): container finished" podID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerID="8b04491a2395287e57cc2cd30c9d9497a85f2b1d3ec6b18255a2ae9f424ef3ef" exitCode=60 Nov 01 20:15:16.342206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.342119 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" event={"ID":"170adf98-24f7-4d09-b0fc-6e31e366b8e1","Type":"ContainerDied","Data":"8b04491a2395287e57cc2cd30c9d9497a85f2b1d3ec6b18255a2ae9f424ef3ef"} Nov 01 20:15:16.342206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.342180 5633 scope.go:122] "RemoveContainer" containerID="ca5a1a44c42d514ab2337f4b160c545b2b0603f9d0f7e53f64e6041cbf0bc60a" Nov 01 20:15:16.342562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.342540 5633 scope.go:122] "RemoveContainer" containerID="8b04491a2395287e57cc2cd30c9d9497a85f2b1d3ec6b18255a2ae9f424ef3ef" Nov 01 20:15:16.342847 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:16.342824 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx_odh-ai-gateway-infra(170adf98-24f7-4d09-b0fc-6e31e366b8e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" Nov 01 20:15:16.451551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.451452 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" podStartSLOduration=16.451438674 podStartE2EDuration="16.451438674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:16.449143644 +0000 UTC m=+724.123402902" watchObservedRunningTime="2026-11-01 20:15:16.451438674 +0000 UTC m=+724.125697929" Nov 01 20:15:16.477199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.477164 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9"] Nov 01 20:15:16.590194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.590163 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx"] Nov 01 20:15:16.651830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.651789 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:16.682587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.682530 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" podStartSLOduration=16.682515417 podStartE2EDuration="16.682515417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:16.681589142 +0000 UTC m=+724.355848397" watchObservedRunningTime="2026-11-01 20:15:16.682515417 +0000 UTC m=+724.356774671" Nov 01 20:15:16.734571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.734541 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b07853c-178d-4f4b-92f4-3040eaa79f90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b07853c-178d-4f4b-92f4-3040eaa79f90-maas-api-service-ca\") pod \"4b07853c-178d-4f4b-92f4-3040eaa79f90\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " Nov 01 20:15:16.734736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.734594 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b07853c-178d-4f4b-92f4-3040eaa79f90-kube-api-access-qw98k\" (UniqueName: \"kubernetes.io/projected/4b07853c-178d-4f4b-92f4-3040eaa79f90-kube-api-access-qw98k\") pod \"4b07853c-178d-4f4b-92f4-3040eaa79f90\" (UID: \"4b07853c-178d-4f4b-92f4-3040eaa79f90\") " Nov 01 20:15:16.734934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.734906 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b07853c-178d-4f4b-92f4-3040eaa79f90-maas-api-service-ca" pod "4b07853c-178d-4f4b-92f4-3040eaa79f90" (UID: "4b07853c-178d-4f4b-92f4-3040eaa79f90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:16.735073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.734998 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b07853c-178d-4f4b-92f4-3040eaa79f90-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:16.736821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.736804 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b07853c-178d-4f4b-92f4-3040eaa79f90-kube-api-access-qw98k" pod "4b07853c-178d-4f4b-92f4-3040eaa79f90" (UID: "4b07853c-178d-4f4b-92f4-3040eaa79f90"). InnerVolumeSpecName "kube-api-access-qw98k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:16.836478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:16.836443 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw98k\" (UniqueName: \"kubernetes.io/projected/4b07853c-178d-4f4b-92f4-3040eaa79f90-kube-api-access-qw98k\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:17.350538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.350505 5633 generic.go:356] "Generic (PLEG): container finished" podID="003def20-e339-4b32-a68c-1b28e7980024" containerID="6a659b5bcaf150d14762e4607a30552745bf7cec97d2c0098f5fc5f06026ac88" exitCode=60 Nov 01 20:15:17.350745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.350577 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" event={"ID":"003def20-e339-4b32-a68c-1b28e7980024","Type":"ContainerDied","Data":"6a659b5bcaf150d14762e4607a30552745bf7cec97d2c0098f5fc5f06026ac88"} Nov 01 20:15:17.350745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.350640 5633 scope.go:122] "RemoveContainer" containerID="e32489f760b35a7418ead5dec8ce42241249323d7756462681a6f270e5b878cd" Nov 01 20:15:17.350997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.350968 5633 scope.go:122] "RemoveContainer" containerID="6a659b5bcaf150d14762e4607a30552745bf7cec97d2c0098f5fc5f06026ac88" Nov 01 20:15:17.351293 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:17.351262 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl_odh-ai-gateway-infra(003def20-e339-4b32-a68c-1b28e7980024)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" podUID="003def20-e339-4b32-a68c-1b28e7980024" Nov 01 20:15:17.352503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.352483 5633 generic.go:356] "Generic (PLEG): container finished" podID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerID="055581f816db1a90dab4b14765f06b9ca28ecc59c98ecf718856774f37ee8243" exitCode=60 Nov 01 20:15:17.352615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.352561 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" event={"ID":"4d241298-cea0-40fc-87c0-6fddbbbe76f3","Type":"ContainerDied","Data":"055581f816db1a90dab4b14765f06b9ca28ecc59c98ecf718856774f37ee8243"} Nov 01 20:15:17.352921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.352905 5633 scope.go:122] "RemoveContainer" containerID="055581f816db1a90dab4b14765f06b9ca28ecc59c98ecf718856774f37ee8243" Nov 01 20:15:17.353189 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:17.353163 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892735-msdjz_odh-ai-gateway-infra(4d241298-cea0-40fc-87c0-6fddbbbe76f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" Nov 01 20:15:17.353919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.353905 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" Nov 01 20:15:17.354034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.354004 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9" event={"ID":"4b07853c-178d-4f4b-92f4-3040eaa79f90","Type":"ContainerDied","Data":"eee35d4ebba63bba4738df362ee7f561e3ca4d525684507631dfa3649d3d9689"} Nov 01 20:15:17.365768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.365746 5633 scope.go:122] "RemoveContainer" containerID="f841dad1e2e01e552abce7c6565937d2548d8aec8baf3ba8afddabd31a555604" Nov 01 20:15:17.380856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.380829 5633 scope.go:122] "RemoveContainer" containerID="beac0881f522440b7dcf440a8070cd122cac1353d08edab418d63e26339dbb95" Nov 01 20:15:17.464362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.464063 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl"] Nov 01 20:15:17.540639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.540609 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:17.619745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.619662 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9"] Nov 01 20:15:17.624433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.624392 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892735-cxqn9"] Nov 01 20:15:17.647351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.647306 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170adf98-24f7-4d09-b0fc-6e31e366b8e1-kube-api-access-vt4pb\" (UniqueName: \"kubernetes.io/projected/170adf98-24f7-4d09-b0fc-6e31e366b8e1-kube-api-access-vt4pb\") pod \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " Nov 01 20:15:17.647506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.647363 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170adf98-24f7-4d09-b0fc-6e31e366b8e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170adf98-24f7-4d09-b0fc-6e31e366b8e1-maas-api-service-ca\") pod \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\" (UID: \"170adf98-24f7-4d09-b0fc-6e31e366b8e1\") " Nov 01 20:15:17.647741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.647715 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170adf98-24f7-4d09-b0fc-6e31e366b8e1-maas-api-service-ca" pod "170adf98-24f7-4d09-b0fc-6e31e366b8e1" (UID: "170adf98-24f7-4d09-b0fc-6e31e366b8e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:17.649686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.649663 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170adf98-24f7-4d09-b0fc-6e31e366b8e1-kube-api-access-vt4pb" pod "170adf98-24f7-4d09-b0fc-6e31e366b8e1" (UID: "170adf98-24f7-4d09-b0fc-6e31e366b8e1"). InnerVolumeSpecName "kube-api-access-vt4pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:17.707118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.707081 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz"] Nov 01 20:15:17.748653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.748611 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt4pb\" (UniqueName: \"kubernetes.io/projected/170adf98-24f7-4d09-b0fc-6e31e366b8e1-kube-api-access-vt4pb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:17.748653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:17.748643 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/170adf98-24f7-4d09-b0fc-6e31e366b8e1-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:18.367100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.367071 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" Nov 01 20:15:18.367268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.367113 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx" event={"ID":"170adf98-24f7-4d09-b0fc-6e31e366b8e1","Type":"ContainerDied","Data":"70a1d8efc1795aa12df9a38054eecb7cff16c930aa3aae96319164d3d25c5312"} Nov 01 20:15:18.367268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.367182 5633 scope.go:122] "RemoveContainer" containerID="8b04491a2395287e57cc2cd30c9d9497a85f2b1d3ec6b18255a2ae9f424ef3ef" Nov 01 20:15:18.474767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.474732 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" path="/var/lib/kubelet/pods/4b07853c-178d-4f4b-92f4-3040eaa79f90/volumes" Nov 01 20:15:18.569550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.569525 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:18.571762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.571742 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:18.660062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.659976 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003def20-e339-4b32-a68c-1b28e7980024-kube-api-access-6h4wf\" (UniqueName: \"kubernetes.io/projected/003def20-e339-4b32-a68c-1b28e7980024-kube-api-access-6h4wf\") pod \"003def20-e339-4b32-a68c-1b28e7980024\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " Nov 01 20:15:18.660062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.660017 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d241298-cea0-40fc-87c0-6fddbbbe76f3-kube-api-access-8dgj2\" (UniqueName: \"kubernetes.io/projected/4d241298-cea0-40fc-87c0-6fddbbbe76f3-kube-api-access-8dgj2\") pod \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " Nov 01 20:15:18.660062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.660047 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003def20-e339-4b32-a68c-1b28e7980024-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003def20-e339-4b32-a68c-1b28e7980024-maas-api-service-ca\") pod \"003def20-e339-4b32-a68c-1b28e7980024\" (UID: \"003def20-e339-4b32-a68c-1b28e7980024\") " Nov 01 20:15:18.660621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.660167 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d241298-cea0-40fc-87c0-6fddbbbe76f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d241298-cea0-40fc-87c0-6fddbbbe76f3-maas-api-service-ca\") pod \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\" (UID: \"4d241298-cea0-40fc-87c0-6fddbbbe76f3\") " Nov 01 20:15:18.660621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.660471 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003def20-e339-4b32-a68c-1b28e7980024-maas-api-service-ca" pod "003def20-e339-4b32-a68c-1b28e7980024" (UID: "003def20-e339-4b32-a68c-1b28e7980024"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:18.660621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.660528 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d241298-cea0-40fc-87c0-6fddbbbe76f3-maas-api-service-ca" pod "4d241298-cea0-40fc-87c0-6fddbbbe76f3" (UID: "4d241298-cea0-40fc-87c0-6fddbbbe76f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:18.662465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.662443 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003def20-e339-4b32-a68c-1b28e7980024-kube-api-access-6h4wf" pod "003def20-e339-4b32-a68c-1b28e7980024" (UID: "003def20-e339-4b32-a68c-1b28e7980024"). InnerVolumeSpecName "kube-api-access-6h4wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:18.662525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.662508 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d241298-cea0-40fc-87c0-6fddbbbe76f3-kube-api-access-8dgj2" pod "4d241298-cea0-40fc-87c0-6fddbbbe76f3" (UID: "4d241298-cea0-40fc-87c0-6fddbbbe76f3"). InnerVolumeSpecName "kube-api-access-8dgj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:18.756825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.756790 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx"] Nov 01 20:15:18.761147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.761113 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h4wf\" (UniqueName: \"kubernetes.io/projected/003def20-e339-4b32-a68c-1b28e7980024-kube-api-access-6h4wf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:18.761147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.761140 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dgj2\" (UniqueName: \"kubernetes.io/projected/4d241298-cea0-40fc-87c0-6fddbbbe76f3-kube-api-access-8dgj2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:18.761147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.761150 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003def20-e339-4b32-a68c-1b28e7980024-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:18.761721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.761159 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d241298-cea0-40fc-87c0-6fddbbbe76f3-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:18.763252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:18.763231 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892735-c9nhx"] Nov 01 20:15:19.380924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.380874 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" event={"ID":"003def20-e339-4b32-a68c-1b28e7980024","Type":"ContainerDied","Data":"9cb66b1c2502e3929cfcd3248451a0ebf8808c913ccb5a22d5930a747c08d91d"} Nov 01 20:15:19.381097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.380895 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl" Nov 01 20:15:19.381097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.380976 5633 scope.go:122] "RemoveContainer" containerID="6a659b5bcaf150d14762e4607a30552745bf7cec97d2c0098f5fc5f06026ac88" Nov 01 20:15:19.382577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.382550 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" Nov 01 20:15:19.382782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.382569 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz" event={"ID":"4d241298-cea0-40fc-87c0-6fddbbbe76f3","Type":"ContainerDied","Data":"5a7b0f8c73f40c8f488c5c124d9049a4d613581e3951bf3dbb07e63e42736f50"} Nov 01 20:15:19.394513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.394486 5633 scope.go:122] "RemoveContainer" containerID="055581f816db1a90dab4b14765f06b9ca28ecc59c98ecf718856774f37ee8243" Nov 01 20:15:19.543944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.543907 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl"] Nov 01 20:15:19.549194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.549166 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892735-4vdwl"] Nov 01 20:15:19.657345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.657238 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz"] Nov 01 20:15:19.669008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.668976 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-msdjz"] Nov 01 20:15:19.991901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:19.991867 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:20.469053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:20.469011 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003def20-e339-4b32-a68c-1b28e7980024" path="/var/lib/kubelet/pods/003def20-e339-4b32-a68c-1b28e7980024/volumes" Nov 01 20:15:20.469723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:20.469701 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" path="/var/lib/kubelet/pods/170adf98-24f7-4d09-b0fc-6e31e366b8e1/volumes" Nov 01 20:15:20.470143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:20.470128 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" path="/var/lib/kubelet/pods/4d241298-cea0-40fc-87c0-6fddbbbe76f3/volumes" Nov 01 20:15:21.295650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:21.295567 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:24.811384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:24.810617 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:27.074427 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.074384 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp"] Nov 01 20:15:27.075111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075088 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075113 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075132 5633 cpu_manager.go:414] "containerMap: removing container" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075139 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075156 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075164 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075184 5633 cpu_manager.go:414] "containerMap: removing container" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075191 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075205 5633 cpu_manager.go:414] "containerMap: removing container" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.075213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075215 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075226 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075234 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075247 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075254 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075273 5633 cpu_manager.go:414] "containerMap: removing container" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075280 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075303 5633 cpu_manager.go:414] "containerMap: removing container" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075310 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075346 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075353 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075366 5633 cpu_manager.go:414] "containerMap: removing container" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075373 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075521 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075537 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075546 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075554 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075563 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075575 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075582 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075594 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:27.075713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.075605 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:27.079040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.079019 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.128251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.128216 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed715511-af16-4acb-9666-91a640c3e535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.128251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.128251 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed715511-af16-4acb-9666-91a640c3e535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.128514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.128284 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8fgm\" (UniqueName: \"kubernetes.io/projected/ed715511-af16-4acb-9666-91a640c3e535-kube-api-access-d8fgm\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.134004 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.133975 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp"] Nov 01 20:15:27.180416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.180385 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:15:27.180687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.180533 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:15:27.180815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.180798 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:15:27.180910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.180893 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:15:27.229263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.229226 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed715511-af16-4acb-9666-91a640c3e535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.229263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.229266 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed715511-af16-4acb-9666-91a640c3e535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.229558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.229298 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8fgm\" (UniqueName: \"kubernetes.io/projected/ed715511-af16-4acb-9666-91a640c3e535-kube-api-access-d8fgm\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.230018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.229991 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed715511-af16-4acb-9666-91a640c3e535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.231896 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.231877 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed715511-af16-4acb-9666-91a640c3e535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.305357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.305037 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8fgm\" (UniqueName: \"kubernetes.io/projected/ed715511-af16-4acb-9666-91a640c3e535-kube-api-access-d8fgm\") pod \"maas-api-revoke-sub-7eb0aa14-sb6rp\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.414527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.414442 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:27.812960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:27.812834 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp"] Nov 01 20:15:27.814307 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:27.814272 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded715511_af16_4acb_9666_91a640c3e535.slice/crio-6ae200e1b743f8a8364447c3293097f4871756f69c8c61ffae2987c52b9bc021 WatchSource:0}: Error finding container 6ae200e1b743f8a8364447c3293097f4871756f69c8c61ffae2987c52b9bc021: Status 404 returned error can't find the container with id 6ae200e1b743f8a8364447c3293097f4871756f69c8c61ffae2987c52b9bc021 Nov 01 20:15:28.474917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:28.474816 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" event={"ID":"ed715511-af16-4acb-9666-91a640c3e535","Type":"ContainerStarted","Data":"fddcfd3fa3f3f3c686632db4311408deefb67bd6194884e2b0f9de684bb476b0"} Nov 01 20:15:28.474917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:28.474888 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" event={"ID":"ed715511-af16-4acb-9666-91a640c3e535","Type":"ContainerStarted","Data":"6ae200e1b743f8a8364447c3293097f4871756f69c8c61ffae2987c52b9bc021"} Nov 01 20:15:28.594127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:28.594060 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" podStartSLOduration=1.594041013 podStartE2EDuration="1.594041013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:28.590048273 +0000 UTC m=+736.264307529" watchObservedRunningTime="2026-11-01 20:15:28.594041013 +0000 UTC m=+736.268300268" Nov 01 20:15:29.482459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:29.482426 5633 generic.go:356] "Generic (PLEG): container finished" podID="ed715511-af16-4acb-9666-91a640c3e535" containerID="fddcfd3fa3f3f3c686632db4311408deefb67bd6194884e2b0f9de684bb476b0" exitCode=0 Nov 01 20:15:29.482920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:29.482510 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" event={"ID":"ed715511-af16-4acb-9666-91a640c3e535","Type":"ContainerDied","Data":"fddcfd3fa3f3f3c686632db4311408deefb67bd6194884e2b0f9de684bb476b0"} Nov 01 20:15:29.636841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:29.636803 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:30.651245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.651210 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:30.665854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.665813 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed715511-af16-4acb-9666-91a640c3e535-kube-api-access-d8fgm\" (UniqueName: \"kubernetes.io/projected/ed715511-af16-4acb-9666-91a640c3e535-kube-api-access-d8fgm\") pod \"ed715511-af16-4acb-9666-91a640c3e535\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " Nov 01 20:15:30.666030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.665888 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed715511-af16-4acb-9666-91a640c3e535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed715511-af16-4acb-9666-91a640c3e535-maas-api-xks-ca\") pod \"ed715511-af16-4acb-9666-91a640c3e535\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " Nov 01 20:15:30.666149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.666127 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed715511-af16-4acb-9666-91a640c3e535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed715511-af16-4acb-9666-91a640c3e535-maas-api-openshift-ca\") pod \"ed715511-af16-4acb-9666-91a640c3e535\" (UID: \"ed715511-af16-4acb-9666-91a640c3e535\") " Nov 01 20:15:30.666740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.666703 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed715511-af16-4acb-9666-91a640c3e535-maas-api-openshift-ca" pod "ed715511-af16-4acb-9666-91a640c3e535" (UID: "ed715511-af16-4acb-9666-91a640c3e535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:30.668174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.668146 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed715511-af16-4acb-9666-91a640c3e535-maas-api-xks-ca" pod "ed715511-af16-4acb-9666-91a640c3e535" (UID: "ed715511-af16-4acb-9666-91a640c3e535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:15:30.668297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.668278 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed715511-af16-4acb-9666-91a640c3e535-kube-api-access-d8fgm" pod "ed715511-af16-4acb-9666-91a640c3e535" (UID: "ed715511-af16-4acb-9666-91a640c3e535"). InnerVolumeSpecName "kube-api-access-d8fgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:30.767623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.767488 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed715511-af16-4acb-9666-91a640c3e535-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:30.767623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.767527 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8fgm\" (UniqueName: \"kubernetes.io/projected/ed715511-af16-4acb-9666-91a640c3e535-kube-api-access-d8fgm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:30.767623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:30.767556 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed715511-af16-4acb-9666-91a640c3e535-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:31.497714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:31.497679 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" event={"ID":"ed715511-af16-4acb-9666-91a640c3e535","Type":"ContainerDied","Data":"6ae200e1b743f8a8364447c3293097f4871756f69c8c61ffae2987c52b9bc021"} Nov 01 20:15:31.497893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:31.497728 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae200e1b743f8a8364447c3293097f4871756f69c8c61ffae2987c52b9bc021" Nov 01 20:15:31.497893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:31.497744 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp" Nov 01 20:15:32.989784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.989754 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4"] Nov 01 20:15:32.990320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990302 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ed715511-af16-4acb-9666-91a640c3e535" containerName="revoke-keys" Nov 01 20:15:32.990384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990322 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed715511-af16-4acb-9666-91a640c3e535" containerName="revoke-keys" Nov 01 20:15:32.990384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990375 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:32.990384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990380 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b07853c-178d-4f4b-92f4-3040eaa79f90" containerName="cleanup" Nov 01 20:15:32.990478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990461 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d241298-cea0-40fc-87c0-6fddbbbe76f3" containerName="cleanup" Nov 01 20:15:32.990478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990473 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed715511-af16-4acb-9666-91a640c3e535" containerName="revoke-keys" Nov 01 20:15:32.990538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990480 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="170adf98-24f7-4d09-b0fc-6e31e366b8e1" containerName="cleanup" Nov 01 20:15:32.990538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.990487 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="003def20-e339-4b32-a68c-1b28e7980024" containerName="cleanup" Nov 01 20:15:32.993942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:32.993925 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.073060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.073009 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:33.092562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.092533 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tls-certs\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.092761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.092648 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-home\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.092761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.092676 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tmp-dir\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.092890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.092771 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-model-cache\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.092890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.092873 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-dshm\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.093001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.092956 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxbrr\" (UniqueName: \"kubernetes.io/projected/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-kube-api-access-jxbrr\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.095747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.095724 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"default-dockercfg-t4w2r\"" Nov 01 20:15:33.095747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.095730 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"kube-root-ca.crt\"" Nov 01 20:15:33.095899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.095734 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"e2e-conflict-b-b637b059-kserve-self-signed-certs\"" Nov 01 20:15:33.095899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.095864 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"openshift-service-ca.crt\"" Nov 01 20:15:33.098639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.098618 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4"] Nov 01 20:15:33.194138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194102 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tls-certs\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194158 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-home\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194180 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tmp-dir\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194222 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-model-cache\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194300 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-dshm\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194401 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxbrr\" (UniqueName: \"kubernetes.io/projected/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-kube-api-access-jxbrr\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194602 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-home\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194622 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tmp-dir\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.194731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.194675 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-model-cache\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.196714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.196691 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-dshm\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.197003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.196984 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tls-certs\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.255509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.255429 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxbrr\" (UniqueName: \"kubernetes.io/projected/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-kube-api-access-jxbrr\") pod \"e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.306680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.306645 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:33.512307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.512205 5633 generic.go:356] "Generic (PLEG): container finished" podID="41cd787f-33ca-49ad-968b-d86b755b5826" containerID="5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274" exitCode=0 Nov 01 20:15:33.512307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.512277 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" event={"ID":"41cd787f-33ca-49ad-968b-d86b755b5826","Type":"ContainerDied","Data":"5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274"} Nov 01 20:15:33.512722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.512707 5633 scope.go:122] "RemoveContainer" containerID="5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274" Nov 01 20:15:33.514158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.514124 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:15:33.514272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.514195 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:15:33.514272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.514233 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:15:33.742978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:33.742952 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4"] Nov 01 20:15:33.746159 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:33.746128 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb864bd27_6cd1_45a5_a5c8_71f7b3ff6e06.slice/crio-8cb9b4ed04c2972672aaa1de385f148ff89d570f452862662fe049ab7f4e4f84 WatchSource:0}: Error finding container 8cb9b4ed04c2972672aaa1de385f148ff89d570f452862662fe049ab7f4e4f84: Status 404 returned error can't find the container with id 8cb9b4ed04c2972672aaa1de385f148ff89d570f452862662fe049ab7f4e4f84 Nov 01 20:15:34.211551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.211289 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:34.336228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.336195 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:34.457190 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.457157 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:34.520979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.520894 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" event={"ID":"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06","Type":"ContainerStarted","Data":"6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd"} Nov 01 20:15:34.520979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.520938 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" event={"ID":"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06","Type":"ContainerStarted","Data":"8cb9b4ed04c2972672aaa1de385f148ff89d570f452862662fe049ab7f4e4f84"} Nov 01 20:15:34.523367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.523319 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" event={"ID":"41cd787f-33ca-49ad-968b-d86b755b5826","Type":"ContainerStarted","Data":"41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f"} Nov 01 20:15:34.582187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.582145 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:34.634522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:34.634472 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" podStartSLOduration=2.634457688 podStartE2EDuration="2.634457688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:34.633537311 +0000 UTC m=+742.307796591" watchObservedRunningTime="2026-11-01 20:15:34.634457688 +0000 UTC m=+742.308716943" Nov 01 20:15:35.308498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:35.308463 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:35.427408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:35.427369 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:15:35.428477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:35.428444 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:15:35.428617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:35.428492 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 20:15:36.428275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:36.428239 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:15:36.428695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:36.428305 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 20:15:37.427959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:37.427923 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:15:37.428171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:37.427986 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 20:15:38.427951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:38.427915 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:15:38.428370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:38.427983 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 20:15:38.951851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:38.951815 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:15:39.098364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:39.085530 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:15:39.406358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:39.404364 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:39.430711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:39.430675 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:15:39.431187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:39.430733 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 20:15:40.433914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:40.433864 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:15:40.435448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:40.435406 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:15:43.307254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:43.307215 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:43.320034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:43.320006 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:43.601735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:43.601646 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:15:44.109817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:44.109784 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:44.977641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:44.977603 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2"] Nov 01 20:15:44.981187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:44.981160 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.071928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.071892 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2"] Nov 01 20:15:45.076741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.076708 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.076908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.076757 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpsc\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-kube-api-access-6tpsc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.076908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.076787 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.077025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.076918 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.077085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.077070 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.077210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.077183 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.077282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.077241 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.077419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.077397 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.077519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.077466 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179208 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179264 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpsc\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-kube-api-access-6tpsc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179285 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179381 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179515 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179597 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179648 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179754 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179774 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.179856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179809 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.180006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.179939 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.180137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.180106 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.180137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.180132 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.180297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.180268 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.181951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.181926 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.182106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.182088 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.236008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.235920 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.236008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.235996 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpsc\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-kube-api-access-6tpsc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.294154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.294117 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:45.731025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.730991 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2"] Nov 01 20:15:45.732772 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:45.732742 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437e5a55_57f6_4a8b_a2b0_893259e41ad3.slice/crio-074657a89933be22eeda57c78f5193c665cb345654c3ce51f730b69fb2d424c7 WatchSource:0}: Error finding container 074657a89933be22eeda57c78f5193c665cb345654c3ce51f730b69fb2d424c7: Status 404 returned error can't find the container with id 074657a89933be22eeda57c78f5193c665cb345654c3ce51f730b69fb2d424c7 Nov 01 20:15:45.735506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.735464 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:15:45.735618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.735545 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:15:45.735618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:45.735593 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:15:46.162039 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:15:46.161949 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:15:46.613869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.613827 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz"] Nov 01 20:15:46.617576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.617550 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.618652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.618625 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" event={"ID":"437e5a55-57f6-4a8b-a2b0-893259e41ad3","Type":"ContainerStarted","Data":"22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088"} Nov 01 20:15:46.618797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.618676 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" event={"ID":"437e5a55-57f6-4a8b-a2b0-893259e41ad3","Type":"ContainerStarted","Data":"074657a89933be22eeda57c78f5193c665cb345654c3ce51f730b69fb2d424c7"} Nov 01 20:15:46.657746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.657690 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz"] Nov 01 20:15:46.710400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.710303 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:15:46.711406 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.710544 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:15:46.711608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.710544 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:15:46.714424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.714398 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.714756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.714714 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.714821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.714794 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfr4t\" (UniqueName: \"kubernetes.io/projected/91977758-d2a6-4f4b-a853-1111f5e4ade9-kube-api-access-wfr4t\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.719618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.719599 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:15:46.815815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.815762 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfr4t\" (UniqueName: \"kubernetes.io/projected/91977758-d2a6-4f4b-a853-1111f5e4ade9-kube-api-access-wfr4t\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.816034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.815913 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.816034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.815975 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.816754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.816728 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.818663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.818638 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.848864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.848800 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podStartSLOduration=2.848777278 podStartE2EDuration="2.848777278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:46.844693077 +0000 UTC m=+754.518952335" watchObservedRunningTime="2026-11-01 20:15:46.848777278 +0000 UTC m=+754.523036534" Nov 01 20:15:46.874246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.874171 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfr4t\" (UniqueName: \"kubernetes.io/projected/91977758-d2a6-4f4b-a853-1111f5e4ade9-kube-api-access-wfr4t\") pod \"maas-api-revoke-sub-899d0e66-bc5sz\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:46.932961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:46.932924 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:47.310378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:47.308157 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:15:47.310378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:47.308222 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 20:15:47.310378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:47.308384 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:47.333021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:47.332988 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz"] Nov 01 20:15:47.334658 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:15:47.334628 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91977758_d2a6_4f4b_a853_1111f5e4ade9.slice/crio-898c2a87f741ff43dc624106b200f86009d4e7544d7a331f0d3254f6797ff597 WatchSource:0}: Error finding container 898c2a87f741ff43dc624106b200f86009d4e7544d7a331f0d3254f6797ff597: Status 404 returned error can't find the container with id 898c2a87f741ff43dc624106b200f86009d4e7544d7a331f0d3254f6797ff597 Nov 01 20:15:47.626838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:47.626745 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" event={"ID":"91977758-d2a6-4f4b-a853-1111f5e4ade9","Type":"ContainerStarted","Data":"898c2a87f741ff43dc624106b200f86009d4e7544d7a331f0d3254f6797ff597"} Nov 01 20:15:48.304182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:48.303727 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:15:48.304182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:48.303786 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 20:15:48.635156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:48.635074 5633 generic.go:356] "Generic (PLEG): container finished" podID="91977758-d2a6-4f4b-a853-1111f5e4ade9" containerID="527852c529d046bc69db8971b7baccf65500ad75e91d420f5ef03da4080fad9f" exitCode=0 Nov 01 20:15:48.636036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:48.636001 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" event={"ID":"91977758-d2a6-4f4b-a853-1111f5e4ade9","Type":"ContainerDied","Data":"527852c529d046bc69db8971b7baccf65500ad75e91d420f5ef03da4080fad9f"} Nov 01 20:15:48.921312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:48.921226 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:49.301364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.301223 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:15:49.301364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.301283 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 20:15:49.825040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.825016 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:49.955130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.954627 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-openshift-ca\") pod \"91977758-d2a6-4f4b-a853-1111f5e4ade9\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " Nov 01 20:15:49.955130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.954708 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91977758-d2a6-4f4b-a853-1111f5e4ade9-kube-api-access-wfr4t\" (UniqueName: \"kubernetes.io/projected/91977758-d2a6-4f4b-a853-1111f5e4ade9-kube-api-access-wfr4t\") pod \"91977758-d2a6-4f4b-a853-1111f5e4ade9\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " Nov 01 20:15:49.955130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.954822 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-xks-ca\") pod \"91977758-d2a6-4f4b-a853-1111f5e4ade9\" (UID: \"91977758-d2a6-4f4b-a853-1111f5e4ade9\") " Nov 01 20:15:49.955130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.955041 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-openshift-ca" pod "91977758-d2a6-4f4b-a853-1111f5e4ade9" (UID: "91977758-d2a6-4f4b-a853-1111f5e4ade9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:49.955481 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.955302 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:49.961353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.961289 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91977758-d2a6-4f4b-a853-1111f5e4ade9-kube-api-access-wfr4t" pod "91977758-d2a6-4f4b-a853-1111f5e4ade9" (UID: "91977758-d2a6-4f4b-a853-1111f5e4ade9"). InnerVolumeSpecName "kube-api-access-wfr4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:49.962782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:49.962751 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-xks-ca" pod "91977758-d2a6-4f4b-a853-1111f5e4ade9" (UID: "91977758-d2a6-4f4b-a853-1111f5e4ade9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:15:50.056344 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:50.056309 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfr4t\" (UniqueName: \"kubernetes.io/projected/91977758-d2a6-4f4b-a853-1111f5e4ade9-kube-api-access-wfr4t\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:50.056539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:50.056369 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91977758-d2a6-4f4b-a853-1111f5e4ade9-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:50.299854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:50.299809 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:15:50.300926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:50.300890 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 20:15:50.648560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:50.648463 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" Nov 01 20:15:50.648560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:50.648495 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz" event={"ID":"91977758-d2a6-4f4b-a853-1111f5e4ade9","Type":"ContainerDied","Data":"898c2a87f741ff43dc624106b200f86009d4e7544d7a331f0d3254f6797ff597"} Nov 01 20:15:50.648748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:50.648586 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898c2a87f741ff43dc624106b200f86009d4e7544d7a331f0d3254f6797ff597" Nov 01 20:15:51.303865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:51.303823 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:15:51.304557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:51.304511 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 20:15:52.294888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:52.294852 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:15:52.295103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:52.294906 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 20:15:54.132099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:54.132064 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:54.132634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:54.132607 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:54.133673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:54.133652 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:15:54.224639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:54.224598 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:56.969199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:56.969161 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:15:59.200191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:15:59.200139 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:01.750387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:01.750350 5633 generic.go:356] "Generic (PLEG): container finished" podID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerID="bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569" exitCode=0 Nov 01 20:16:01.750836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:01.750397 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" event={"ID":"113e6f55-ced3-4c66-b51d-83f8bc007233","Type":"ContainerDied","Data":"bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569"} Nov 01 20:16:01.750836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:01.750786 5633 scope.go:122] "RemoveContainer" containerID="bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569" Nov 01 20:16:01.752293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:01.752262 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:16:01.752393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:01.752353 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:16:01.752438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:01.752396 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:16:02.760074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:02.760038 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" event={"ID":"113e6f55-ced3-4c66-b51d-83f8bc007233","Type":"ContainerStarted","Data":"2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba"} Nov 01 20:16:02.784200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:02.784168 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:16:03.473668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:03.473632 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:16:03.473858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:03.473811 5633 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:16:03.473932 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:03.473864 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 20:16:04.119776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.119724 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:04.474898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.474857 5633 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:16:04.475115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.474913 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 20:16:04.672981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.672947 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5"] Nov 01 20:16:04.673571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.673557 5633 cpu_manager.go:414] "containerMap: removing container" podUID="91977758-d2a6-4f4b-a853-1111f5e4ade9" containerName="revoke-keys" Nov 01 20:16:04.673623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.673573 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91977758-d2a6-4f4b-a853-1111f5e4ade9" containerName="revoke-keys" Nov 01 20:16:04.673676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.673667 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="91977758-d2a6-4f4b-a853-1111f5e4ade9" containerName="revoke-keys" Nov 01 20:16:04.677238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.677217 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.733189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.733090 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5"] Nov 01 20:16:04.779034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.779005 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:16:04.779034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.779022 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:16:04.779034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.779008 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:16:04.779034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.779045 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:16:04.782681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.782657 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.782822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.782718 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.782892 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.782845 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xgf\" (UniqueName: \"kubernetes.io/projected/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-kube-api-access-g5xgf\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.884098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.884062 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xgf\" (UniqueName: \"kubernetes.io/projected/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-kube-api-access-g5xgf\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.884287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.884226 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.884287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.884273 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.884949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.884926 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.902040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.901999 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:04.942534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.942496 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xgf\" (UniqueName: \"kubernetes.io/projected/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-kube-api-access-g5xgf\") pod \"maas-api-revoke-sub-273ab7d8-27rp5\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:05.004414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:04.994538 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:05.415473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:05.415434 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5"] Nov 01 20:16:05.416100 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:16:05.416072 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcb2d15_cee3_4418_ad33_7ca7d7f0c256.slice/crio-481c485b0a17295d12f942071ca07ff414ae8769c332edcc68475872f673589f WatchSource:0}: Error finding container 481c485b0a17295d12f942071ca07ff414ae8769c332edcc68475872f673589f: Status 404 returned error can't find the container with id 481c485b0a17295d12f942071ca07ff414ae8769c332edcc68475872f673589f Nov 01 20:16:05.474235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:05.474195 5633 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:16:05.474424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:05.474246 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 20:16:05.782893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:05.782854 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" event={"ID":"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256","Type":"ContainerStarted","Data":"803e11dbb7a8bdda1d9dfc3e316e7a3a6948860fb0bd725f59ca5f22e1be428c"} Nov 01 20:16:05.783074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:05.782940 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" event={"ID":"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256","Type":"ContainerStarted","Data":"481c485b0a17295d12f942071ca07ff414ae8769c332edcc68475872f673589f"} Nov 01 20:16:06.473829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:06.473796 5633 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:16:06.474260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:06.473854 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 20:16:06.812787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:06.812736 5633 generic.go:356] "Generic (PLEG): container finished" podID="5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" containerID="803e11dbb7a8bdda1d9dfc3e316e7a3a6948860fb0bd725f59ca5f22e1be428c" exitCode=0 Nov 01 20:16:06.812998 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:06.812834 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" event={"ID":"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256","Type":"ContainerDied","Data":"803e11dbb7a8bdda1d9dfc3e316e7a3a6948860fb0bd725f59ca5f22e1be428c"} Nov 01 20:16:07.474288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:07.474252 5633 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:16:07.474719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:07.474304 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 20:16:07.682606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:07.682563 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:07.980028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:07.979990 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:08.041466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.041436 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-xks-ca\") pod \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " Nov 01 20:16:08.041663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.041499 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-kube-api-access-g5xgf\" (UniqueName: \"kubernetes.io/projected/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-kube-api-access-g5xgf\") pod \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " Nov 01 20:16:08.041663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.041537 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-openshift-ca\") pod \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\" (UID: \"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256\") " Nov 01 20:16:08.041961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.041926 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-openshift-ca" pod "5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" (UID: "5dcb2d15-cee3-4418-ad33-7ca7d7f0c256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:16:08.042383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.042153 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:08.043835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.043806 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-xks-ca" pod "5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" (UID: "5dcb2d15-cee3-4418-ad33-7ca7d7f0c256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:16:08.043933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.043850 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-kube-api-access-g5xgf" pod "5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" (UID: "5dcb2d15-cee3-4418-ad33-7ca7d7f0c256"). InnerVolumeSpecName "kube-api-access-g5xgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:16:08.143776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.143689 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:08.143776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.143718 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xgf\" (UniqueName: \"kubernetes.io/projected/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256-kube-api-access-g5xgf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:08.479087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.479058 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:16:08.479504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.479435 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:16:08.480095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.480080 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:16:08.757790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.757708 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:08.857834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.857795 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" Nov 01 20:16:08.858041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.857837 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5" event={"ID":"5dcb2d15-cee3-4418-ad33-7ca7d7f0c256","Type":"ContainerDied","Data":"481c485b0a17295d12f942071ca07ff414ae8769c332edcc68475872f673589f"} Nov 01 20:16:08.858041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.857880 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="481c485b0a17295d12f942071ca07ff414ae8769c332edcc68475872f673589f" Nov 01 20:16:08.933154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:08.933100 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:12.746787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:12.746743 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:13.671887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:13.671848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:14.993116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:14.993079 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c"] Nov 01 20:16:14.993645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:14.993632 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" containerName="revoke-keys" Nov 01 20:16:14.993713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:14.993646 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" containerName="revoke-keys" Nov 01 20:16:14.993764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:14.993755 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" containerName="revoke-keys" Nov 01 20:16:14.999080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:14.999045 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.103662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.103620 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c"] Nov 01 20:16:15.135782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.135749 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.135782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.135785 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.135970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.135870 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.136031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.135965 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.136031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.136011 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.136099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.136072 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.136099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.136088 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2817334-701e-455f-9874-0ed3d1a2cc13-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.136158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.136113 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.136195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.136167 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-kube-api-access-p5wsl\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.236917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.236883 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.236927 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.236943 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2817334-701e-455f-9874-0ed3d1a2cc13-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.236973 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.236997 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-kube-api-access-p5wsl\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237083 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237111 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237177 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237266 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237321 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237403 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237610 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237652 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.237822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.237798 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2817334-701e-455f-9874-0ed3d1a2cc13-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.239724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.239704 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.239938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.239919 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.293508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.293439 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.293760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.293736 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-kube-api-access-p5wsl\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.313570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.313536 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:15.743904 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.743874 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c"] Nov 01 20:16:15.744430 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:16:15.744403 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2817334_701e_455f_9874_0ed3d1a2cc13.slice/crio-0561e514f20ee8b7c5c5065e3057134a9f55313f163708a30b9be1cfa48c4938 WatchSource:0}: Error finding container 0561e514f20ee8b7c5c5065e3057134a9f55313f163708a30b9be1cfa48c4938: Status 404 returned error can't find the container with id 0561e514f20ee8b7c5c5065e3057134a9f55313f163708a30b9be1cfa48c4938 Nov 01 20:16:15.746881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.746848 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:16:15.746983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.746912 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:16:15.746983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.746941 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:16:15.940478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.939760 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" event={"ID":"a2817334-701e-455f-9874-0ed3d1a2cc13","Type":"ContainerStarted","Data":"e9946db3dfc77055b8a3f9b187508b432ff74b13450324aa341e5b75b0a7f9f7"} Nov 01 20:16:15.940974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:15.940947 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" event={"ID":"a2817334-701e-455f-9874-0ed3d1a2cc13","Type":"ContainerStarted","Data":"0561e514f20ee8b7c5c5065e3057134a9f55313f163708a30b9be1cfa48c4938"} Nov 01 20:16:16.072576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:16.072444 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podStartSLOduration=2.072423931 podStartE2EDuration="2.072423931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:16.068109315 +0000 UTC m=+783.742368573" watchObservedRunningTime="2026-11-01 20:16:16.072423931 +0000 UTC m=+783.746683186" Nov 01 20:16:16.180151 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:16:16.179143 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:16:16.318357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:16.318096 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:16.318357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:16.318227 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:16:16.318357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:16.318268 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 20:16:17.316359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:17.315776 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:16:17.316359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:17.315846 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 20:16:18.314827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:18.314782 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:16:18.314995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:18.314867 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 20:16:18.514213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:18.514164 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:19.316032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:19.315990 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:16:19.316240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:19.316073 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 20:16:20.314924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:20.314883 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:16:20.315470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:20.314952 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 20:16:20.360009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:20.359969 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:20.811289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:20.811233 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:21.316413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:21.316323 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:16:21.317309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:21.317277 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 20:16:21.512716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:21.508020 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:21.637445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:21.637352 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:22.314010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:22.313971 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 20:16:22.314191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:22.314036 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 20:16:23.319268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:23.319230 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:23.319748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:23.319544 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:23.320203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:23.320186 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:16:23.594974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:23.594888 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:25.458023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:25.457984 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:25.641896 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:25.641863 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:25.775722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:25.775692 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:27.086063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:27.086029 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:16:28.675944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:28.675895 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:28.845950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:28.845903 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:33.692968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:33.692929 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:34.736250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:34.736199 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4"] Nov 01 20:16:34.742110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:34.742063 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:36.603435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.603390 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb"] Nov 01 20:16:36.607815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.607791 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.663602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.663567 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb"] Nov 01 20:16:36.682648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.682612 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.682852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.682810 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.682852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.682845 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxpw\" (UniqueName: \"kubernetes.io/projected/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-kube-api-access-cnxpw\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.709013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.708974 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:16:36.709013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.708971 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:16:36.709210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.709023 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:16:36.710130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.710118 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:16:36.784395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.784324 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.784605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.784491 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.784605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.784512 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxpw\" (UniqueName: \"kubernetes.io/projected/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-kube-api-access-cnxpw\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.785179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.785153 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.786916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.786891 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.841012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.840972 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxpw\" (UniqueName: \"kubernetes.io/projected/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-kube-api-access-cnxpw\") pod \"maas-api-revoke-sub-64a1b989-4wlbb\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:36.923038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:36.922946 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:37.303052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:37.303030 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb"] Nov 01 20:16:37.304123 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:16:37.304089 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode941ee79_7eb1_4511_b7f2_705c3f04c7a7.slice/crio-9d410ffce36a74ec7de7ad492937703c7b23cc15ab2f83c741fe8af704c0a882 WatchSource:0}: Error finding container 9d410ffce36a74ec7de7ad492937703c7b23cc15ab2f83c741fe8af704c0a882: Status 404 returned error can't find the container with id 9d410ffce36a74ec7de7ad492937703c7b23cc15ab2f83c741fe8af704c0a882 Nov 01 20:16:38.147758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:38.147720 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" event={"ID":"e941ee79-7eb1-4511-b7f2-705c3f04c7a7","Type":"ContainerStarted","Data":"7c00df074166dcae3f5179a949d3818f7cc1068342c2672af2ef3dd4d4cde606"} Nov 01 20:16:38.148168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:38.147796 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" event={"ID":"e941ee79-7eb1-4511-b7f2-705c3f04c7a7","Type":"ContainerStarted","Data":"9d410ffce36a74ec7de7ad492937703c7b23cc15ab2f83c741fe8af704c0a882"} Nov 01 20:16:38.259961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:38.259893 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" podStartSLOduration=2.259876488 podStartE2EDuration="2.259876488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:38.259593547 +0000 UTC m=+805.933852803" watchObservedRunningTime="2026-11-01 20:16:38.259876488 +0000 UTC m=+805.934135755" Nov 01 20:16:38.394497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:38.394456 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:39.002650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:39.002608 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:39.156072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:39.156033 5633 generic.go:356] "Generic (PLEG): container finished" podID="e941ee79-7eb1-4511-b7f2-705c3f04c7a7" containerID="7c00df074166dcae3f5179a949d3818f7cc1068342c2672af2ef3dd4d4cde606" exitCode=0 Nov 01 20:16:39.156588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:39.156117 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" event={"ID":"e941ee79-7eb1-4511-b7f2-705c3f04c7a7","Type":"ContainerDied","Data":"7c00df074166dcae3f5179a949d3818f7cc1068342c2672af2ef3dd4d4cde606"} Nov 01 20:16:40.323870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.323840 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:40.430365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.430260 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-kube-api-access-cnxpw\" (UniqueName: \"kubernetes.io/projected/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-kube-api-access-cnxpw\") pod \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " Nov 01 20:16:40.430365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.430358 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-openshift-ca\") pod \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " Nov 01 20:16:40.430599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.430475 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-xks-ca\") pod \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\" (UID: \"e941ee79-7eb1-4511-b7f2-705c3f04c7a7\") " Nov 01 20:16:40.430713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.430691 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-openshift-ca" pod "e941ee79-7eb1-4511-b7f2-705c3f04c7a7" (UID: "e941ee79-7eb1-4511-b7f2-705c3f04c7a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:16:40.430956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.430938 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:40.432575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.432555 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-xks-ca" pod "e941ee79-7eb1-4511-b7f2-705c3f04c7a7" (UID: "e941ee79-7eb1-4511-b7f2-705c3f04c7a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:16:40.432696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.432673 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-kube-api-access-cnxpw" pod "e941ee79-7eb1-4511-b7f2-705c3f04c7a7" (UID: "e941ee79-7eb1-4511-b7f2-705c3f04c7a7"). InnerVolumeSpecName "kube-api-access-cnxpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:16:40.532265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.532225 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnxpw\" (UniqueName: \"kubernetes.io/projected/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-kube-api-access-cnxpw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:40.532265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:40.532256 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e941ee79-7eb1-4511-b7f2-705c3f04c7a7-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:41.175353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:41.175270 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" event={"ID":"e941ee79-7eb1-4511-b7f2-705c3f04c7a7","Type":"ContainerDied","Data":"9d410ffce36a74ec7de7ad492937703c7b23cc15ab2f83c741fe8af704c0a882"} Nov 01 20:16:41.175559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:41.175358 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb" Nov 01 20:16:41.175559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:41.175379 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d410ffce36a74ec7de7ad492937703c7b23cc15ab2f83c741fe8af704c0a882" Nov 01 20:16:43.325532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:43.325500 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:46.162523 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:16:46.162488 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:16:47.980471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:47.980432 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:49.200808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.200774 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl"] Nov 01 20:16:49.203251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.203222 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e941ee79-7eb1-4511-b7f2-705c3f04c7a7" containerName="revoke-keys" Nov 01 20:16:49.203501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.203485 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e941ee79-7eb1-4511-b7f2-705c3f04c7a7" containerName="revoke-keys" Nov 01 20:16:49.203763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.203750 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e941ee79-7eb1-4511-b7f2-705c3f04c7a7" containerName="revoke-keys" Nov 01 20:16:49.208786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.208762 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.260099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.260063 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl"] Nov 01 20:16:49.304744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.304710 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:16:49.304898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.304741 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:16:49.304898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.304769 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:16:49.309838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.309793 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:16:49.373184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.373140 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.373184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.373189 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8j5\" (UniqueName: \"kubernetes.io/projected/09f50230-bb2d-4a1f-a2ef-7f1361480592-kube-api-access-nd8j5\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.373493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.373375 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.474128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.474090 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.474376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.474142 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8j5\" (UniqueName: \"kubernetes.io/projected/09f50230-bb2d-4a1f-a2ef-7f1361480592-kube-api-access-nd8j5\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.474376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.474244 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.475060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.475035 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.477068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.477038 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.530553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.530513 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8j5\" (UniqueName: \"kubernetes.io/projected/09f50230-bb2d-4a1f-a2ef-7f1361480592-kube-api-access-nd8j5\") pod \"maas-api-revoke-sub-fb2ab432-ch5vl\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.550385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.550320 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:49.714555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.714510 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:49.758524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.758445 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" podUID="b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" containerName="main" containerID="cri-o://6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd" gracePeriod=45 Nov 01 20:16:49.862298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.862238 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:49.934450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:49.934308 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl"] Nov 01 20:16:49.935500 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:16:49.935390 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f50230_bb2d_4a1f_a2ef_7f1361480592.slice/crio-5edbe28ae1e80c9cac23647cc2b1b3d86280cd43f68040d3057087845c889ce3 WatchSource:0}: Error finding container 5edbe28ae1e80c9cac23647cc2b1b3d86280cd43f68040d3057087845c889ce3: Status 404 returned error can't find the container with id 5edbe28ae1e80c9cac23647cc2b1b3d86280cd43f68040d3057087845c889ce3 Nov 01 20:16:50.027382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.027350 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:16:50.045533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.045495 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:50.190973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.190936 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-home\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-home\") pod \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " Nov 01 20:16:50.191161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.190998 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tls-certs\" (UniqueName: \"kubernetes.io/secret/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tls-certs\") pod \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " Nov 01 20:16:50.191161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191075 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-kube-api-access-jxbrr\" (UniqueName: \"kubernetes.io/projected/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-kube-api-access-jxbrr\") pod \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " Nov 01 20:16:50.191282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191216 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-dshm\") pod \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " Nov 01 20:16:50.191282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191259 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-home" pod "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" (UID: "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:50.191394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191309 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-model-cache\") pod \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " Nov 01 20:16:50.191443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191429 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tmp-dir\") pod \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\" (UID: \"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06\") " Nov 01 20:16:50.191538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191516 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-model-cache" pod "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" (UID: "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:50.191685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191664 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tmp-dir" pod "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" (UID: "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:50.191924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191907 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:50.191981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191930 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:50.191981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.191943 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:50.193467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.193442 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tls-certs" pod "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" (UID: "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:16:50.193562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.193496 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-dshm" pod "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" (UID: "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:50.193562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.193536 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-kube-api-access-jxbrr" pod "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" (UID: "b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06"). InnerVolumeSpecName "kube-api-access-jxbrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:16:50.245409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.245369 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" event={"ID":"09f50230-bb2d-4a1f-a2ef-7f1361480592","Type":"ContainerStarted","Data":"0b471a8f46d72aab9d0f2f9187557f4e73bbbbe93b69ed1635dd87ddca874181"} Nov 01 20:16:50.245858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.245440 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" event={"ID":"09f50230-bb2d-4a1f-a2ef-7f1361480592","Type":"ContainerStarted","Data":"5edbe28ae1e80c9cac23647cc2b1b3d86280cd43f68040d3057087845c889ce3"} Nov 01 20:16:50.246983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.246956 5633 generic.go:356] "Generic (PLEG): container finished" podID="b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" containerID="6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd" exitCode=0 Nov 01 20:16:50.247108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.246986 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" Nov 01 20:16:50.247108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.247058 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" event={"ID":"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06","Type":"ContainerDied","Data":"6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd"} Nov 01 20:16:50.247108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.247094 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4" event={"ID":"b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06","Type":"ContainerDied","Data":"8cb9b4ed04c2972672aaa1de385f148ff89d570f452862662fe049ab7f4e4f84"} Nov 01 20:16:50.247268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.247116 5633 scope.go:122] "RemoveContainer" containerID="6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd" Nov 01 20:16:50.257490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.257466 5633 scope.go:122] "RemoveContainer" containerID="6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd" Nov 01 20:16:50.257816 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:16:50.257794 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd\": container with ID starting with 6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd not found: ID does not exist" containerID="6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd" Nov 01 20:16:50.257897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.257827 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd"} err="failed to get container status \"6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd\": rpc error: code = NotFound desc = could not find container \"6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd\": container with ID starting with 6b15c40fae955f914d700dd991b2556b2194d14be17b8f4eccf8a0abfa54eefd not found: ID does not exist" Nov 01 20:16:50.293437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.293364 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:50.293437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.293389 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxbrr\" (UniqueName: \"kubernetes.io/projected/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-kube-api-access-jxbrr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:50.293437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.293408 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:50.359681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.359624 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" podStartSLOduration=1.359608398 podStartE2EDuration="1.359608398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:50.356955274 +0000 UTC m=+818.031214530" watchObservedRunningTime="2026-11-01 20:16:50.359608398 +0000 UTC m=+818.033867692" Nov 01 20:16:50.467970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.467941 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4"] Nov 01 20:16:50.472882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.472852 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-b-b637b059-kserve-c6c4449b7-r2qj4"] Nov 01 20:16:50.966065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:50.966000 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:51.095484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.095356 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:51.256003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.255971 5633 generic.go:356] "Generic (PLEG): container finished" podID="09f50230-bb2d-4a1f-a2ef-7f1361480592" containerID="0b471a8f46d72aab9d0f2f9187557f4e73bbbbe93b69ed1635dd87ddca874181" exitCode=0 Nov 01 20:16:51.256481 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.256057 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" event={"ID":"09f50230-bb2d-4a1f-a2ef-7f1361480592","Type":"ContainerDied","Data":"0b471a8f46d72aab9d0f2f9187557f4e73bbbbe93b69ed1635dd87ddca874181"} Nov 01 20:16:51.927777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.927739 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb"] Nov 01 20:16:51.928315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.928302 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" containerName="main" Nov 01 20:16:51.928389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.928317 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" containerName="main" Nov 01 20:16:51.928471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.928461 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" containerName="main" Nov 01 20:16:51.932025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:51.932002 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.018794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.018742 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb"] Nov 01 20:16:52.026984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.026941 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9bs\" (UniqueName: \"kubernetes.io/projected/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-kube-api-access-9k9bs\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.027184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.027006 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.027184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.027073 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.128762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.128728 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9bs\" (UniqueName: \"kubernetes.io/projected/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-kube-api-access-9k9bs\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.128762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.128771 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.129019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.128816 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.129573 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.129551 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.131237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.131219 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.185308 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.185237 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9bs\" (UniqueName: \"kubernetes.io/projected/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-kube-api-access-9k9bs\") pod \"maas-api-revoke-sub-b737cb6c-2sswb\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.245599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.245562 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:52.468349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.468300 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06" path="/var/lib/kubelet/pods/b864bd27-6cd1-45a5-a5c8-71f7b3ff6e06/volumes" Nov 01 20:16:52.478864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.478836 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:52.634263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.634227 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-openshift-ca\") pod \"09f50230-bb2d-4a1f-a2ef-7f1361480592\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " Nov 01 20:16:52.634479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.634295 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-xks-ca\") pod \"09f50230-bb2d-4a1f-a2ef-7f1361480592\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " Nov 01 20:16:52.634479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.634471 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f50230-bb2d-4a1f-a2ef-7f1361480592-kube-api-access-nd8j5\" (UniqueName: \"kubernetes.io/projected/09f50230-bb2d-4a1f-a2ef-7f1361480592-kube-api-access-nd8j5\") pod \"09f50230-bb2d-4a1f-a2ef-7f1361480592\" (UID: \"09f50230-bb2d-4a1f-a2ef-7f1361480592\") " Nov 01 20:16:52.634685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.634657 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-openshift-ca" pod "09f50230-bb2d-4a1f-a2ef-7f1361480592" (UID: "09f50230-bb2d-4a1f-a2ef-7f1361480592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:16:52.635020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.634995 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:52.636706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.636684 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f50230-bb2d-4a1f-a2ef-7f1361480592-kube-api-access-nd8j5" pod "09f50230-bb2d-4a1f-a2ef-7f1361480592" (UID: "09f50230-bb2d-4a1f-a2ef-7f1361480592"). InnerVolumeSpecName "kube-api-access-nd8j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:16:52.636706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.636692 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-xks-ca" pod "09f50230-bb2d-4a1f-a2ef-7f1361480592" (UID: "09f50230-bb2d-4a1f-a2ef-7f1361480592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:16:52.689497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.689473 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb"] Nov 01 20:16:52.691389 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:16:52.691322 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c7c6a6_d853_48e8_ba56_db17fff6f46b.slice/crio-90db98368d29b6d9f62b2d67365c2243174ec29f3f0c96135f37d59ebf1e0022 WatchSource:0}: Error finding container 90db98368d29b6d9f62b2d67365c2243174ec29f3f0c96135f37d59ebf1e0022: Status 404 returned error can't find the container with id 90db98368d29b6d9f62b2d67365c2243174ec29f3f0c96135f37d59ebf1e0022 Nov 01 20:16:52.735756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.735730 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50230-bb2d-4a1f-a2ef-7f1361480592-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:52.735756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:52.735757 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd8j5\" (UniqueName: \"kubernetes.io/projected/09f50230-bb2d-4a1f-a2ef-7f1361480592-kube-api-access-nd8j5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:53.068104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:53.068069 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:53.272037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:53.271927 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" event={"ID":"b6c7c6a6-d853-48e8-ba56-db17fff6f46b","Type":"ContainerStarted","Data":"ec4af858e5ceafa3f18a6e4ad654a127e8b66db2432173964780043e5038e81f"} Nov 01 20:16:53.272037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:53.271989 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" event={"ID":"b6c7c6a6-d853-48e8-ba56-db17fff6f46b","Type":"ContainerStarted","Data":"90db98368d29b6d9f62b2d67365c2243174ec29f3f0c96135f37d59ebf1e0022"} Nov 01 20:16:53.273580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:53.273549 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" event={"ID":"09f50230-bb2d-4a1f-a2ef-7f1361480592","Type":"ContainerDied","Data":"5edbe28ae1e80c9cac23647cc2b1b3d86280cd43f68040d3057087845c889ce3"} Nov 01 20:16:53.273696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:53.273590 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl" Nov 01 20:16:53.273696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:53.273598 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5edbe28ae1e80c9cac23647cc2b1b3d86280cd43f68040d3057087845c889ce3" Nov 01 20:16:53.386984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:53.386932 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" podStartSLOduration=2.3869174060000002 podStartE2EDuration="2.386917406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:53.384745361 +0000 UTC m=+821.059004617" watchObservedRunningTime="2026-11-01 20:16:53.386917406 +0000 UTC m=+821.061176663" Nov 01 20:16:54.281678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:54.281645 5633 generic.go:356] "Generic (PLEG): container finished" podID="b6c7c6a6-d853-48e8-ba56-db17fff6f46b" containerID="ec4af858e5ceafa3f18a6e4ad654a127e8b66db2432173964780043e5038e81f" exitCode=0 Nov 01 20:16:54.282075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:54.281730 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" event={"ID":"b6c7c6a6-d853-48e8-ba56-db17fff6f46b","Type":"ContainerDied","Data":"ec4af858e5ceafa3f18a6e4ad654a127e8b66db2432173964780043e5038e81f"} Nov 01 20:16:55.447533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.447502 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:55.574049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.573967 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-openshift-ca\") pod \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " Nov 01 20:16:55.574214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.574044 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-xks-ca\") pod \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " Nov 01 20:16:55.574214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.574090 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-kube-api-access-9k9bs\" (UniqueName: \"kubernetes.io/projected/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-kube-api-access-9k9bs\") pod \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\" (UID: \"b6c7c6a6-d853-48e8-ba56-db17fff6f46b\") " Nov 01 20:16:55.574410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.574382 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-openshift-ca" pod "b6c7c6a6-d853-48e8-ba56-db17fff6f46b" (UID: "b6c7c6a6-d853-48e8-ba56-db17fff6f46b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:16:55.574768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.574748 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:55.576988 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.576961 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-kube-api-access-9k9bs" pod "b6c7c6a6-d853-48e8-ba56-db17fff6f46b" (UID: "b6c7c6a6-d853-48e8-ba56-db17fff6f46b"). InnerVolumeSpecName "kube-api-access-9k9bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:16:55.577295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.577274 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-xks-ca" pod "b6c7c6a6-d853-48e8-ba56-db17fff6f46b" (UID: "b6c7c6a6-d853-48e8-ba56-db17fff6f46b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:16:55.583028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.582996 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:16:55.676633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.676598 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:55.676633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:55.676629 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k9bs\" (UniqueName: \"kubernetes.io/projected/b6c7c6a6-d853-48e8-ba56-db17fff6f46b-kube-api-access-9k9bs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:56.297288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:56.297258 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" Nov 01 20:16:56.297517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:56.297296 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb" event={"ID":"b6c7c6a6-d853-48e8-ba56-db17fff6f46b","Type":"ContainerDied","Data":"90db98368d29b6d9f62b2d67365c2243174ec29f3f0c96135f37d59ebf1e0022"} Nov 01 20:16:56.297517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:56.297383 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90db98368d29b6d9f62b2d67365c2243174ec29f3f0c96135f37d59ebf1e0022" Nov 01 20:16:56.704001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:56.703916 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:16:58.056294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:16:58.056181 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:01.305917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:01.305874 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:02.735566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:02.735531 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:02.906363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:02.906308 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:03.211559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:03.211521 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:06.735711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:06.735669 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:07.393933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.393897 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4"] Nov 01 20:17:07.394515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.394501 5633 cpu_manager.go:414] "containerMap: removing container" podUID="09f50230-bb2d-4a1f-a2ef-7f1361480592" containerName="revoke-keys" Nov 01 20:17:07.394563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.394517 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f50230-bb2d-4a1f-a2ef-7f1361480592" containerName="revoke-keys" Nov 01 20:17:07.394563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.394549 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b6c7c6a6-d853-48e8-ba56-db17fff6f46b" containerName="revoke-keys" Nov 01 20:17:07.394563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.394554 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c7c6a6-d853-48e8-ba56-db17fff6f46b" containerName="revoke-keys" Nov 01 20:17:07.394658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.394637 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c7c6a6-d853-48e8-ba56-db17fff6f46b" containerName="revoke-keys" Nov 01 20:17:07.394658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.394645 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f50230-bb2d-4a1f-a2ef-7f1361480592" containerName="revoke-keys" Nov 01 20:17:07.398393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.398373 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.453306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.453273 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4"] Nov 01 20:17:07.473617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.473373 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxn8d\" (UniqueName: \"kubernetes.io/projected/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-kube-api-access-sxn8d\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.473617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.473440 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.473846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.473690 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.499266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.499223 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:17:07.499266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.499261 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:17:07.500214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.500185 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:17:07.500214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.500185 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:17:07.574974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.574939 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.575173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.575148 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxn8d\" (UniqueName: \"kubernetes.io/projected/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-kube-api-access-sxn8d\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.575257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.575198 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.575808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.575788 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.577568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.577546 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.631875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.631839 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxn8d\" (UniqueName: \"kubernetes.io/projected/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-kube-api-access-sxn8d\") pod \"maas-api-revoke-sub-4a87a329-rwlx4\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:07.667766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.667669 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:07.712806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:07.712764 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:08.106014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:08.105975 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4"] Nov 01 20:17:08.107199 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:17:08.107168 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ec2944_b62c_467b_8f03_7bccc4abc4fc.slice/crio-7e7956fabcdf2e0d4e8bad62980e21af24bdbc832fdd5b92501c6c289ff2f1e2 WatchSource:0}: Error finding container 7e7956fabcdf2e0d4e8bad62980e21af24bdbc832fdd5b92501c6c289ff2f1e2: Status 404 returned error can't find the container with id 7e7956fabcdf2e0d4e8bad62980e21af24bdbc832fdd5b92501c6c289ff2f1e2 Nov 01 20:17:08.427407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:08.427370 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" event={"ID":"c3ec2944-b62c-467b-8f03-7bccc4abc4fc","Type":"ContainerStarted","Data":"1f95d0100d6a9e94e464857a26ee3670d7847b04ca918f5d78b5f573b6ad12f0"} Nov 01 20:17:08.427593 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:08.427464 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" event={"ID":"c3ec2944-b62c-467b-8f03-7bccc4abc4fc","Type":"ContainerStarted","Data":"7e7956fabcdf2e0d4e8bad62980e21af24bdbc832fdd5b92501c6c289ff2f1e2"} Nov 01 20:17:08.539946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:08.539840 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" podStartSLOduration=1.53982078 podStartE2EDuration="1.53982078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:08.539004443 +0000 UTC m=+836.213263699" watchObservedRunningTime="2026-11-01 20:17:08.53982078 +0000 UTC m=+836.214080036" Nov 01 20:17:09.435507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:09.435419 5633 generic.go:356] "Generic (PLEG): container finished" podID="c3ec2944-b62c-467b-8f03-7bccc4abc4fc" containerID="1f95d0100d6a9e94e464857a26ee3670d7847b04ca918f5d78b5f573b6ad12f0" exitCode=0 Nov 01 20:17:09.435871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:09.435515 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" event={"ID":"c3ec2944-b62c-467b-8f03-7bccc4abc4fc","Type":"ContainerDied","Data":"1f95d0100d6a9e94e464857a26ee3670d7847b04ca918f5d78b5f573b6ad12f0"} Nov 01 20:17:10.587062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.587036 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:10.695747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.695653 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x"] Nov 01 20:17:10.696735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.696710 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c3ec2944-b62c-467b-8f03-7bccc4abc4fc" containerName="revoke-keys" Nov 01 20:17:10.696809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.696738 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ec2944-b62c-467b-8f03-7bccc4abc4fc" containerName="revoke-keys" Nov 01 20:17:10.696905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.696892 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ec2944-b62c-467b-8f03-7bccc4abc4fc" containerName="revoke-keys" Nov 01 20:17:10.700959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.700926 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.714852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.714820 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-openshift-ca\") pod \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " Nov 01 20:17:10.715016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.714891 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-kube-api-access-sxn8d\" (UniqueName: \"kubernetes.io/projected/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-kube-api-access-sxn8d\") pod \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " Nov 01 20:17:10.715078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.715060 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-xks-ca\") pod \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\" (UID: \"c3ec2944-b62c-467b-8f03-7bccc4abc4fc\") " Nov 01 20:17:10.715321 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.715210 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-openshift-ca" pod "c3ec2944-b62c-467b-8f03-7bccc4abc4fc" (UID: "c3ec2944-b62c-467b-8f03-7bccc4abc4fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:10.715567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.715546 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:10.717753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.717724 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-kube-api-access-sxn8d" pod "c3ec2944-b62c-467b-8f03-7bccc4abc4fc" (UID: "c3ec2944-b62c-467b-8f03-7bccc4abc4fc"). InnerVolumeSpecName "kube-api-access-sxn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:17:10.717894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.717876 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-xks-ca" pod "c3ec2944-b62c-467b-8f03-7bccc4abc4fc" (UID: "c3ec2944-b62c-467b-8f03-7bccc4abc4fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:17:10.733282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.733227 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:10.743268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.743239 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x"] Nov 01 20:17:10.753350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.753292 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"kube-root-ca.crt\"" Nov 01 20:17:10.753350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.753292 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"openshift-service-ca.crt\"" Nov 01 20:17:10.753558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.753485 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"default-dockercfg-t4w2r\"" Nov 01 20:17:10.753558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.753550 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"distinct-w3-1b1134-kserve-self-signed-certs\"" Nov 01 20:17:10.818929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.817183 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdw8k\" (UniqueName: \"kubernetes.io/projected/b4f1de7c-e045-4fb6-98e4-97972982358c-kube-api-access-tdw8k\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.818929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.818450 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-home\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.818929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.818616 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f1de7c-e045-4fb6-98e4-97972982358c-tls-certs\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.818929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.818931 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-dshm\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.819264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.818987 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-tmp-dir\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.819372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.819268 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-model-cache\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.819534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.819515 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:10.819714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.819542 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxn8d\" (UniqueName: \"kubernetes.io/projected/c3ec2944-b62c-467b-8f03-7bccc4abc4fc-kube-api-access-sxn8d\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:10.920150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920104 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdw8k\" (UniqueName: \"kubernetes.io/projected/b4f1de7c-e045-4fb6-98e4-97972982358c-kube-api-access-tdw8k\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.920385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920159 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-home\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.920385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920196 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f1de7c-e045-4fb6-98e4-97972982358c-tls-certs\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.920385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920303 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-dshm\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.920544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920468 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-tmp-dir\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.920588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920558 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-model-cache\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.920839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920622 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-home\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.920909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.920851 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-tmp-dir\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.921025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.921004 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-model-cache\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.922949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.922927 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-dshm\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.923040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.923023 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f1de7c-e045-4fb6-98e4-97972982358c-tls-certs\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:10.981805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:10.981774 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdw8k\" (UniqueName: \"kubernetes.io/projected/b4f1de7c-e045-4fb6-98e4-97972982358c-kube-api-access-tdw8k\") pod \"distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:11.018887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:11.018849 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:11.395742 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:11.395710 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x"] Nov 01 20:17:11.397295 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:17:11.397263 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4f1de7c_e045_4fb6_98e4_97972982358c.slice/crio-3764a7503b21d00ff33777ab79a40fead515ad25db3ed038c66744a17f9868f5 WatchSource:0}: Error finding container 3764a7503b21d00ff33777ab79a40fead515ad25db3ed038c66744a17f9868f5: Status 404 returned error can't find the container with id 3764a7503b21d00ff33777ab79a40fead515ad25db3ed038c66744a17f9868f5 Nov 01 20:17:11.450105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:11.450081 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" Nov 01 20:17:11.450245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:11.450080 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4" event={"ID":"c3ec2944-b62c-467b-8f03-7bccc4abc4fc","Type":"ContainerDied","Data":"7e7956fabcdf2e0d4e8bad62980e21af24bdbc832fdd5b92501c6c289ff2f1e2"} Nov 01 20:17:11.450245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:11.450215 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e7956fabcdf2e0d4e8bad62980e21af24bdbc832fdd5b92501c6c289ff2f1e2" Nov 01 20:17:11.456697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:11.456611 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" event={"ID":"b4f1de7c-e045-4fb6-98e4-97972982358c","Type":"ContainerStarted","Data":"3764a7503b21d00ff33777ab79a40fead515ad25db3ed038c66744a17f9868f5"} Nov 01 20:17:12.402967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:12.402934 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:12.484361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:12.484309 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" event={"ID":"b4f1de7c-e045-4fb6-98e4-97972982358c","Type":"ContainerStarted","Data":"577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898"} Nov 01 20:17:12.604430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:12.604369 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podStartSLOduration=2.6043520989999998 podStartE2EDuration="2.604352099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:12.603595092 +0000 UTC m=+840.277854348" watchObservedRunningTime="2026-11-01 20:17:12.604352099 +0000 UTC m=+840.278611352" Nov 01 20:17:13.018997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:13.018958 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:14.979477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:14.979435 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7"] Nov 01 20:17:14.985890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:14.985859 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.109558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.109526 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.109766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.109579 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.109766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.109599 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.109885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.109772 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n5rs\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-kube-api-access-8n5rs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.109885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.109828 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.109885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.109861 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.110072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.110057 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.110135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.110118 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.110189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.110174 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.182091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.182052 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7"] Nov 01 20:17:15.211369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211299 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211419 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211442 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211470 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211518 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211540 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211587 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n5rs\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-kube-api-access-8n5rs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211629 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211653 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.211970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.211946 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.212259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.212232 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.212400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.212374 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.212746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.212726 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.212829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.212760 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.214151 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.214120 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.214625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.214604 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.267888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.267800 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.268155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.268096 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n5rs\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-kube-api-access-8n5rs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.300876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.300801 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:15.732935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.732902 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7"] Nov 01 20:17:15.733522 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:17:15.733494 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21d6128_3b0a_4d41_828c_ed92bf4611a5.slice/crio-42c3cf826021f1ec623b69b983f94a6acea86a68737152b773665f79925d28b2 WatchSource:0}: Error finding container 42c3cf826021f1ec623b69b983f94a6acea86a68737152b773665f79925d28b2: Status 404 returned error can't find the container with id 42c3cf826021f1ec623b69b983f94a6acea86a68737152b773665f79925d28b2 Nov 01 20:17:15.736041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.735984 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:17:15.736138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.736076 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:17:15.736138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:15.736104 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:17:16.165637 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:17:16.165532 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:17:16.516682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:16.516633 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" event={"ID":"f21d6128-3b0a-4d41-828c-ed92bf4611a5","Type":"ContainerStarted","Data":"a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892"} Nov 01 20:17:16.516897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:16.516707 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" event={"ID":"f21d6128-3b0a-4d41-828c-ed92bf4611a5","Type":"ContainerStarted","Data":"42c3cf826021f1ec623b69b983f94a6acea86a68737152b773665f79925d28b2"} Nov 01 20:17:16.733120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:16.733069 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podStartSLOduration=2.733053356 podStartE2EDuration="2.733053356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:16.682139656 +0000 UTC m=+844.356398912" watchObservedRunningTime="2026-11-01 20:17:16.733053356 +0000 UTC m=+844.407312677" Nov 01 20:17:17.224887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:17.224847 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:17.316872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:17.307997 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:17.320528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:17.320491 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 20:17:17.321180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:17.321142 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 20:17:18.112703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:18.112655 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:18.301808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:18.301770 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 20:17:18.302439 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:18.302404 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 20:17:19.302838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:19.302801 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 20:17:19.303471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:19.303432 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 20:17:20.306791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:20.306756 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 20:17:20.307571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:20.307527 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 20:17:21.019736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:21.019702 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:21.032918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:21.032890 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:21.305419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:21.302845 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 20:17:21.305419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:21.302937 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 20:17:21.571067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:21.570990 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:17:22.068020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:22.067981 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:22.199541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:22.199460 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:22.302020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:22.301983 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 20:17:22.302243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:22.302047 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 20:17:22.343446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:22.343316 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:23.306320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:23.306289 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:23.306747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:23.306729 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:23.307519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:23.307497 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:17:26.675167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:26.675133 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:27.423572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.423537 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk"] Nov 01 20:17:27.427749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.427729 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.483731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.483699 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk"] Nov 01 20:17:27.528986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.528950 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:17:27.529185 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.528995 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:17:27.529252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.529189 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:17:27.529252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.529206 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:17:27.599425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.599378 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vc2\" (UniqueName: \"kubernetes.io/projected/f848e91c-1ab9-46d9-9ca9-724557034e45-kube-api-access-v8vc2\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.599615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.599566 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.599667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.599643 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.700612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.700502 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.701062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.700625 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.701062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.700706 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vc2\" (UniqueName: \"kubernetes.io/projected/f848e91c-1ab9-46d9-9ca9-724557034e45-kube-api-access-v8vc2\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.701203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.701181 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.703178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.703157 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:27.759786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:27.759755 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vc2\" (UniqueName: \"kubernetes.io/projected/f848e91c-1ab9-46d9-9ca9-724557034e45-kube-api-access-v8vc2\") pod \"maas-api-revoke-sub-d60f273a-vp8wk\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:28.042415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:28.042381 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:28.418030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:28.418006 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk"] Nov 01 20:17:28.419699 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:17:28.419673 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf848e91c_1ab9_46d9_9ca9_724557034e45.slice/crio-01ffd5b13be2b5642e80e842225b539280279c25a8a399d063bab72268b870f2 WatchSource:0}: Error finding container 01ffd5b13be2b5642e80e842225b539280279c25a8a399d063bab72268b870f2: Status 404 returned error can't find the container with id 01ffd5b13be2b5642e80e842225b539280279c25a8a399d063bab72268b870f2 Nov 01 20:17:28.613546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:28.613506 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" event={"ID":"f848e91c-1ab9-46d9-9ca9-724557034e45","Type":"ContainerStarted","Data":"01ffd5b13be2b5642e80e842225b539280279c25a8a399d063bab72268b870f2"} Nov 01 20:17:29.621756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:29.621676 5633 generic.go:356] "Generic (PLEG): container finished" podID="f848e91c-1ab9-46d9-9ca9-724557034e45" containerID="73106f3246b74a8dbcc5479f7becef239292624f72e199d45332508be41073db" exitCode=0 Nov 01 20:17:29.622126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:29.621766 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" event={"ID":"f848e91c-1ab9-46d9-9ca9-724557034e45","Type":"ContainerDied","Data":"73106f3246b74a8dbcc5479f7becef239292624f72e199d45332508be41073db"} Nov 01 20:17:30.072747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.072701 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s"] Nov 01 20:17:30.077145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.077123 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.181965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.181931 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s"] Nov 01 20:17:30.231912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.231872 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.231963 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.232016 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbc82\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-kube-api-access-fbc82\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.232043 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.232105 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed39b20-1f38-4a11-a398-ccb791e0a159-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.232162 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.232189 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.232276 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.232362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.232344 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333448 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333507 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333569 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333605 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333664 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbc82\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-kube-api-access-fbc82\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333708 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333743 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed39b20-1f38-4a11-a398-ccb791e0a159-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333789 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.333973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.333826 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.334542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.334506 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.334768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.334533 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.334768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.334583 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.334768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.334642 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.335148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.335122 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed39b20-1f38-4a11-a398-ccb791e0a159-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.336628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.336607 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.336859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.336836 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.389980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.389947 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.390214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.390193 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbc82\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-kube-api-access-fbc82\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.392062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.392042 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:30.402012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.401979 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:17:30.858057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.857250 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:17:30.858057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.857360 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:17:30.858057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.857407 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:17:30.862902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.862836 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s"] Nov 01 20:17:30.980050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:30.979702 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:31.165943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.164886 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-xks-ca\") pod \"f848e91c-1ab9-46d9-9ca9-724557034e45\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " Nov 01 20:17:31.165943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.164971 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f848e91c-1ab9-46d9-9ca9-724557034e45-kube-api-access-v8vc2\" (UniqueName: \"kubernetes.io/projected/f848e91c-1ab9-46d9-9ca9-724557034e45-kube-api-access-v8vc2\") pod \"f848e91c-1ab9-46d9-9ca9-724557034e45\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " Nov 01 20:17:31.165943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.165079 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-openshift-ca\") pod \"f848e91c-1ab9-46d9-9ca9-724557034e45\" (UID: \"f848e91c-1ab9-46d9-9ca9-724557034e45\") " Nov 01 20:17:31.166445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.166418 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-openshift-ca" pod "f848e91c-1ab9-46d9-9ca9-724557034e45" (UID: "f848e91c-1ab9-46d9-9ca9-724557034e45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:31.268409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.268362 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:31.296780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.296736 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-xks-ca" pod "f848e91c-1ab9-46d9-9ca9-724557034e45" (UID: "f848e91c-1ab9-46d9-9ca9-724557034e45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:17:31.305839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.305781 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f848e91c-1ab9-46d9-9ca9-724557034e45-kube-api-access-v8vc2" pod "f848e91c-1ab9-46d9-9ca9-724557034e45" (UID: "f848e91c-1ab9-46d9-9ca9-724557034e45"). InnerVolumeSpecName "kube-api-access-v8vc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:17:31.372313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.372248 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848e91c-1ab9-46d9-9ca9-724557034e45-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:31.372313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.372284 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vc2\" (UniqueName: \"kubernetes.io/projected/f848e91c-1ab9-46d9-9ca9-724557034e45-kube-api-access-v8vc2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:31.615396 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.600719 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:31.778271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.778230 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" event={"ID":"f848e91c-1ab9-46d9-9ca9-724557034e45","Type":"ContainerDied","Data":"01ffd5b13be2b5642e80e842225b539280279c25a8a399d063bab72268b870f2"} Nov 01 20:17:31.782356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.779400 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ffd5b13be2b5642e80e842225b539280279c25a8a399d063bab72268b870f2" Nov 01 20:17:31.782356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.779504 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk" Nov 01 20:17:31.847131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.846092 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" event={"ID":"1ed39b20-1f38-4a11-a398-ccb791e0a159","Type":"ContainerStarted","Data":"e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178"} Nov 01 20:17:31.847131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.846163 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" event={"ID":"1ed39b20-1f38-4a11-a398-ccb791e0a159","Type":"ContainerStarted","Data":"5048aa72923deb4e0f797d0ef43aaac558033459785c84935bd114a26fe3aabc"} Nov 01 20:17:31.993356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:31.992406 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podStartSLOduration=1.9923832799999999 podStartE2EDuration="1.99238328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:31.964885564 +0000 UTC m=+859.639144820" watchObservedRunningTime="2026-11-01 20:17:31.99238328 +0000 UTC m=+859.666642536" Nov 01 20:17:32.393357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:32.392734 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:32.403387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:32.396465 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:17:32.403387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:32.396524 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:17:33.409532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:33.407762 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:17:33.409532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:33.407829 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:17:34.405359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:34.403734 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:17:34.405359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:34.403812 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:17:35.410824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:35.410785 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:17:35.411305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:35.410847 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:17:36.410361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:36.395775 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:17:36.410361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:36.395843 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:17:36.609068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:36.609029 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:37.406863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:37.406825 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:17:37.407163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:37.407135 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:17:38.248542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.248492 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62"] Nov 01 20:17:38.249312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.249290 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f848e91c-1ab9-46d9-9ca9-724557034e45" containerName="revoke-keys" Nov 01 20:17:38.249442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.249314 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f848e91c-1ab9-46d9-9ca9-724557034e45" containerName="revoke-keys" Nov 01 20:17:38.249521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.249507 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f848e91c-1ab9-46d9-9ca9-724557034e45" containerName="revoke-keys" Nov 01 20:17:38.253303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.253281 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.308972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.308938 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62"] Nov 01 20:17:38.354517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.354484 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:17:38.354699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.354498 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:17:38.354699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.354541 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:17:38.355009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.354989 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:17:38.393225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.393190 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 20:17:38.393462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.393261 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 20:17:38.401161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.401124 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.402550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.401435 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/e830b283-0b3d-49de-97c2-9a59499bb1de-kube-api-access-6rzv2\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.402550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.401481 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.502623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.502531 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/e830b283-0b3d-49de-97c2-9a59499bb1de-kube-api-access-6rzv2\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.502623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.502581 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.502880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.502692 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.503385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.503362 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.505322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.505300 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.558782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.558754 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/e830b283-0b3d-49de-97c2-9a59499bb1de-kube-api-access-6rzv2\") pod \"maas-api-revoke-sub-7882123d-srh62\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.570939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.570911 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:38.736553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.736513 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw"] Nov 01 20:17:38.742430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.742405 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:38.797114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.797025 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw"] Nov 01 20:17:38.907393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.907356 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:38.907612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.907409 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:38.907612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.907486 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtn22\" (UniqueName: \"kubernetes.io/projected/69e234c5-6fcb-48f9-847c-1aa828c392c3-kube-api-access-jtn22\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:38.967517 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:17:38.967481 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode830b283_0b3d_49de_97c2_9a59499bb1de.slice/crio-e07873de8529b2400c265733eaa342ca03812387d44084bf5a95bfe1fa18546d WatchSource:0}: Error finding container e07873de8529b2400c265733eaa342ca03812387d44084bf5a95bfe1fa18546d: Status 404 returned error can't find the container with id e07873de8529b2400c265733eaa342ca03812387d44084bf5a95bfe1fa18546d Nov 01 20:17:38.968578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:38.968551 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62"] Nov 01 20:17:39.008829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.008795 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:39.008984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.008876 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:39.008984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.008956 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtn22\" (UniqueName: \"kubernetes.io/projected/69e234c5-6fcb-48f9-847c-1aa828c392c3-kube-api-access-jtn22\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:39.009513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.009491 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:39.011406 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.011379 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:39.065531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.065452 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtn22\" (UniqueName: \"kubernetes.io/projected/69e234c5-6fcb-48f9-847c-1aa828c392c3-kube-api-access-jtn22\") pod \"maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:39.359512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.359434 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:39.505505 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.505467 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:39.505915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.505894 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:39.507084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.507060 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:17:39.755647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.755612 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw"] Nov 01 20:17:39.943421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.943377 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" event={"ID":"e830b283-0b3d-49de-97c2-9a59499bb1de","Type":"ContainerStarted","Data":"4ae9abf5d64219e670c2355eaa8d8d6111649c917a31cbc0f373ece6b819f8ff"} Nov 01 20:17:39.943621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.943455 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" event={"ID":"e830b283-0b3d-49de-97c2-9a59499bb1de","Type":"ContainerStarted","Data":"e07873de8529b2400c265733eaa342ca03812387d44084bf5a95bfe1fa18546d"} Nov 01 20:17:39.945049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:39.945018 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" event={"ID":"69e234c5-6fcb-48f9-847c-1aa828c392c3","Type":"ContainerStarted","Data":"1de3ee4618d57059014b44932a78eb1acec27dac19cadc8281ba31c513ea0a1a"} Nov 01 20:17:40.059933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:40.059879 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" podStartSLOduration=2.059860548 podStartE2EDuration="2.059860548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:40.055733368 +0000 UTC m=+867.729992623" watchObservedRunningTime="2026-11-01 20:17:40.059860548 +0000 UTC m=+867.734119830" Nov 01 20:17:40.954045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:40.953949 5633 generic.go:356] "Generic (PLEG): container finished" podID="e830b283-0b3d-49de-97c2-9a59499bb1de" containerID="4ae9abf5d64219e670c2355eaa8d8d6111649c917a31cbc0f373ece6b819f8ff" exitCode=0 Nov 01 20:17:40.954045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:40.954020 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" event={"ID":"e830b283-0b3d-49de-97c2-9a59499bb1de","Type":"ContainerDied","Data":"4ae9abf5d64219e670c2355eaa8d8d6111649c917a31cbc0f373ece6b819f8ff"} Nov 01 20:17:40.955762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:40.955733 5633 generic.go:356] "Generic (PLEG): container finished" podID="69e234c5-6fcb-48f9-847c-1aa828c392c3" containerID="e5e52a3bcf8d87d6ef514a530d4425736f0c8905010450b6ed254a20bc57dc35" exitCode=0 Nov 01 20:17:40.955877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:40.955815 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" event={"ID":"69e234c5-6fcb-48f9-847c-1aa828c392c3","Type":"ContainerDied","Data":"e5e52a3bcf8d87d6ef514a530d4425736f0c8905010450b6ed254a20bc57dc35"} Nov 01 20:17:41.491730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:41.491693 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:42.164392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.164353 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:42.167629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.167604 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:42.351461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.351371 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-openshift-ca\") pod \"e830b283-0b3d-49de-97c2-9a59499bb1de\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " Nov 01 20:17:42.351461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.351426 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-xks-ca\") pod \"69e234c5-6fcb-48f9-847c-1aa828c392c3\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " Nov 01 20:17:42.351683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.351478 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e234c5-6fcb-48f9-847c-1aa828c392c3-kube-api-access-jtn22\" (UniqueName: \"kubernetes.io/projected/69e234c5-6fcb-48f9-847c-1aa828c392c3-kube-api-access-jtn22\") pod \"69e234c5-6fcb-48f9-847c-1aa828c392c3\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " Nov 01 20:17:42.351683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.351520 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-xks-ca\") pod \"e830b283-0b3d-49de-97c2-9a59499bb1de\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " Nov 01 20:17:42.351683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.351559 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-openshift-ca\") pod \"69e234c5-6fcb-48f9-847c-1aa828c392c3\" (UID: \"69e234c5-6fcb-48f9-847c-1aa828c392c3\") " Nov 01 20:17:42.351846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.351698 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e830b283-0b3d-49de-97c2-9a59499bb1de-kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/e830b283-0b3d-49de-97c2-9a59499bb1de-kube-api-access-6rzv2\") pod \"e830b283-0b3d-49de-97c2-9a59499bb1de\" (UID: \"e830b283-0b3d-49de-97c2-9a59499bb1de\") " Nov 01 20:17:42.351846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.351777 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-openshift-ca" pod "e830b283-0b3d-49de-97c2-9a59499bb1de" (UID: "e830b283-0b3d-49de-97c2-9a59499bb1de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:42.352039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.352019 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-openshift-ca" pod "69e234c5-6fcb-48f9-847c-1aa828c392c3" (UID: "69e234c5-6fcb-48f9-847c-1aa828c392c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:42.352301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.352273 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.352301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.352303 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.353957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.353927 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-xks-ca" pod "e830b283-0b3d-49de-97c2-9a59499bb1de" (UID: "e830b283-0b3d-49de-97c2-9a59499bb1de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:17:42.354060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.353993 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e830b283-0b3d-49de-97c2-9a59499bb1de-kube-api-access-6rzv2" pod "e830b283-0b3d-49de-97c2-9a59499bb1de" (UID: "e830b283-0b3d-49de-97c2-9a59499bb1de"). InnerVolumeSpecName "kube-api-access-6rzv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:17:42.354240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.354221 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-xks-ca" pod "69e234c5-6fcb-48f9-847c-1aa828c392c3" (UID: "69e234c5-6fcb-48f9-847c-1aa828c392c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:17:42.354294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.354245 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e234c5-6fcb-48f9-847c-1aa828c392c3-kube-api-access-jtn22" pod "69e234c5-6fcb-48f9-847c-1aa828c392c3" (UID: "69e234c5-6fcb-48f9-847c-1aa828c392c3"). InnerVolumeSpecName "kube-api-access-jtn22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:17:42.453517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.453473 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rzv2\" (UniqueName: \"kubernetes.io/projected/e830b283-0b3d-49de-97c2-9a59499bb1de-kube-api-access-6rzv2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.453517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.453506 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e234c5-6fcb-48f9-847c-1aa828c392c3-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.453517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.453515 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtn22\" (UniqueName: \"kubernetes.io/projected/69e234c5-6fcb-48f9-847c-1aa828c392c3-kube-api-access-jtn22\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.453517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.453526 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e830b283-0b3d-49de-97c2-9a59499bb1de-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.972398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.972357 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" event={"ID":"e830b283-0b3d-49de-97c2-9a59499bb1de","Type":"ContainerDied","Data":"e07873de8529b2400c265733eaa342ca03812387d44084bf5a95bfe1fa18546d"} Nov 01 20:17:42.972398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.972382 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62" Nov 01 20:17:42.972673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.972416 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e07873de8529b2400c265733eaa342ca03812387d44084bf5a95bfe1fa18546d" Nov 01 20:17:42.974116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.974096 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" Nov 01 20:17:42.974243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.974119 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw" event={"ID":"69e234c5-6fcb-48f9-847c-1aa828c392c3","Type":"ContainerDied","Data":"1de3ee4618d57059014b44932a78eb1acec27dac19cadc8281ba31c513ea0a1a"} Nov 01 20:17:42.974243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:42.974162 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de3ee4618d57059014b44932a78eb1acec27dac19cadc8281ba31c513ea0a1a" Nov 01 20:17:46.184130 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:17:46.184084 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:17:46.306362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:46.301763 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:46.703822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:46.703785 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:51.407693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:51.407657 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:52.887952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.887919 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf"] Nov 01 20:17:52.888518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.888503 5633 cpu_manager.go:414] "containerMap: removing container" podUID="69e234c5-6fcb-48f9-847c-1aa828c392c3" containerName="revoke-keys" Nov 01 20:17:52.888582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.888518 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e234c5-6fcb-48f9-847c-1aa828c392c3" containerName="revoke-keys" Nov 01 20:17:52.888582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.888548 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e830b283-0b3d-49de-97c2-9a59499bb1de" containerName="revoke-keys" Nov 01 20:17:52.888582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.888553 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e830b283-0b3d-49de-97c2-9a59499bb1de" containerName="revoke-keys" Nov 01 20:17:52.888680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.888656 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e234c5-6fcb-48f9-847c-1aa828c392c3" containerName="revoke-keys" Nov 01 20:17:52.888680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.888667 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e830b283-0b3d-49de-97c2-9a59499bb1de" containerName="revoke-keys" Nov 01 20:17:52.893691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.893672 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:52.925912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.925862 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-tmp-dir\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:52.926081 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.925983 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-dshm\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:52.926081 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.926030 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-model-cache\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:52.926196 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.926080 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a0661c-0fc6-43b8-8881-1177313e219b-tls-certs\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:52.926292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.926268 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-home\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:52.926405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.926388 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqcx\" (UniqueName: \"kubernetes.io/projected/01a0661c-0fc6-43b8-8881-1177313e219b-kube-api-access-7lqcx\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:52.949036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.949000 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf"] Nov 01 20:17:52.976493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.976443 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:52.993849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.993802 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ad7edb\"/\"openshift-service-ca.crt\"" Nov 01 20:17:52.994030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.993885 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ad7edb\"/\"default-dockercfg-wfst4\"" Nov 01 20:17:52.994030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.993904 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ad7edb\"/\"kube-root-ca.crt\"" Nov 01 20:17:52.995462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:52.995442 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ad7edb\"/\"test-model-ad7edb-kserve-self-signed-certs\"" Nov 01 20:17:53.028106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028076 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqcx\" (UniqueName: \"kubernetes.io/projected/01a0661c-0fc6-43b8-8881-1177313e219b-kube-api-access-7lqcx\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028160 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-tmp-dir\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028222 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-dshm\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028258 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-model-cache\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028301 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a0661c-0fc6-43b8-8881-1177313e219b-tls-certs\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028402 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-home\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028684 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-tmp-dir\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028732 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-model-cache\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.028804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.028785 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-home\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.030845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.030818 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-dshm\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.031132 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.031115 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a0661c-0fc6-43b8-8881-1177313e219b-tls-certs\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.096612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.096475 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqcx\" (UniqueName: \"kubernetes.io/projected/01a0661c-0fc6-43b8-8881-1177313e219b-kube-api-access-7lqcx\") pod \"test-model-ad7edb-kserve-b6966ccb9-ltbqf\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.206438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.206310 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:53.584997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:53.584970 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf"] Nov 01 20:17:53.585636 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:17:53.585606 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01a0661c_0fc6_43b8_8881_1177313e219b.slice/crio-c71e533b136d88734834306358d3cfdfe86d03d19c5ef4b401ecc81725c80b28 WatchSource:0}: Error finding container c71e533b136d88734834306358d3cfdfe86d03d19c5ef4b401ecc81725c80b28: Status 404 returned error can't find the container with id c71e533b136d88734834306358d3cfdfe86d03d19c5ef4b401ecc81725c80b28 Nov 01 20:17:54.086884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:54.086846 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" event={"ID":"01a0661c-0fc6-43b8-8881-1177313e219b","Type":"ContainerStarted","Data":"7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1"} Nov 01 20:17:54.087282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:54.086899 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" event={"ID":"01a0661c-0fc6-43b8-8881-1177313e219b","Type":"ContainerStarted","Data":"c71e533b136d88734834306358d3cfdfe86d03d19c5ef4b401ecc81725c80b28"} Nov 01 20:17:54.203560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:54.203507 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podStartSLOduration=2.203493208 podStartE2EDuration="2.203493208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:54.200463549 +0000 UTC m=+881.874722803" watchObservedRunningTime="2026-11-01 20:17:54.203493208 +0000 UTC m=+881.877752488" Nov 01 20:17:54.207445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:54.207410 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:17:56.028675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:56.028640 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:17:59.986235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:59.986196 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp"] Nov 01 20:17:59.990720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:17:59.990703 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.038094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.038054 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7"] Nov 01 20:18:00.043581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.043552 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.127600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127559 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.127808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127606 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.127808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127654 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9070b4cc-cc17-4396-9a63-83963140bcc8-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.127808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127679 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.127808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127699 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.127808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127741 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.127808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127767 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbwc\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-kube-api-access-chbwc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.127808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127800 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127816 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127830 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127862 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127879 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127895 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8d8\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-kube-api-access-4z8d8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127922 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127941 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.127979 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.128033 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.128228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.128074 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.170169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.170130 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7"] Nov 01 20:18:00.190052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.190015 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp"] Nov 01 20:18:00.229189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229157 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbwc\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-kube-api-access-chbwc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.229389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229204 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229320 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.229389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229389 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229425 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.229549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229440 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229458 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8d8\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-kube-api-access-4z8d8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229539 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229580 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229640 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229695 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.229779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229738 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229775 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229856 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.229985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229889 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.229985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229940 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9070b4cc-cc17-4396-9a63-83963140bcc8-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.229985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229946 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.230312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.229990 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.230312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230037 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.230312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230078 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.230312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230112 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.230312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230177 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.230312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230266 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.230657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230466 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.230754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230731 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9070b4cc-cc17-4396-9a63-83963140bcc8-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.230814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.230757 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.231167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.231143 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.231287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.231262 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.232223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.232190 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.232381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.232304 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.232747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.232724 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.232815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.232797 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.286220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.286143 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.286220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.286156 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.286417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.286301 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbwc\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-kube-api-access-chbwc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.286417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.286308 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8d8\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-kube-api-access-4z8d8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.307562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.307531 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:00.359513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.359477 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:00.709164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.709076 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:00.735955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.735923 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp"] Nov 01 20:18:00.785949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.785877 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:00.786085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.785968 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:00.786085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.786020 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:00.793300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.793268 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7"] Nov 01 20:18:00.793801 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:18:00.793772 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9070b4cc_cc17_4396_9a63_83963140bcc8.slice/crio-f50a3cd29300830f3a74c1485ae9d8228274e759412d0ee508e926f46064321e WatchSource:0}: Error finding container f50a3cd29300830f3a74c1485ae9d8228274e759412d0ee508e926f46064321e: Status 404 returned error can't find the container with id f50a3cd29300830f3a74c1485ae9d8228274e759412d0ee508e926f46064321e Nov 01 20:18:00.801765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.801722 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:00.801929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.801813 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:00.801929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:00.801855 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:01.144374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.144290 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" event={"ID":"9070b4cc-cc17-4396-9a63-83963140bcc8","Type":"ContainerStarted","Data":"309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e"} Nov 01 20:18:01.144901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.144406 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" event={"ID":"9070b4cc-cc17-4396-9a63-83963140bcc8","Type":"ContainerStarted","Data":"f50a3cd29300830f3a74c1485ae9d8228274e759412d0ee508e926f46064321e"} Nov 01 20:18:01.146500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.146470 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" event={"ID":"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed","Type":"ContainerStarted","Data":"7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31"} Nov 01 20:18:01.146654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.146518 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" event={"ID":"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed","Type":"ContainerStarted","Data":"6ac442d5f10be8f19b54f1386c3497c749071839a30b41f222a33651cac7ced6"} Nov 01 20:18:01.262400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.262320 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podStartSLOduration=2.262304344 podStartE2EDuration="2.262304344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:01.258877682 +0000 UTC m=+888.933136938" watchObservedRunningTime="2026-11-01 20:18:01.262304344 +0000 UTC m=+888.936563668" Nov 01 20:18:01.308213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.308167 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:01.309946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.309906 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:01.310110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.309966 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:01.360343 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.360294 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:01.361727 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.361701 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:01.361858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.361750 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:01.376019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:01.375970 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podStartSLOduration=2.375956715 podStartE2EDuration="2.375956715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:01.37326655 +0000 UTC m=+889.047525806" watchObservedRunningTime="2026-11-01 20:18:01.375956715 +0000 UTC m=+889.050215970" Nov 01 20:18:02.309358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:02.308804 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:02.309358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:02.308870 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:02.360644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:02.360607 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:02.360816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:02.360663 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:03.208930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:03.208393 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:18:03.246350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:03.246304 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:18:03.323360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:03.309760 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:03.323360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:03.309822 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:03.359860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:03.359825 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:03.360039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:03.359884 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:03.901359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:03.900396 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:18:04.261145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:04.261114 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:18:04.311371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:04.308770 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:04.311371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:04.308838 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:04.360788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:04.360665 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:04.360788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:04.360730 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:05.335361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:05.327495 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:05.335361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:05.327566 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:05.360780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:05.360743 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:05.360956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:05.360806 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:05.741399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:05.728046 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:06.312821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:06.312779 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:06.313079 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:06.313051 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:06.360114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:06.360082 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:06.360325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:06.360303 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:07.314355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:07.312766 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:07.314355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:07.312833 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:07.360317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:07.360278 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:07.360509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:07.360365 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:08.055612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.054982 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj"] Nov 01 20:18:08.103399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.103358 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24416a5c-5knsj"] Nov 01 20:18:08.308508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.308399 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 20:18:08.308508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.308463 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 20:18:08.360400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.360360 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 20:18:08.360894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.360429 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 20:18:08.477785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.477695 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180210f7-1850-414d-9c2e-5fb3f07f07ee" path="/var/lib/kubelet/pods/180210f7-1850-414d-9c2e-5fb3f07f07ee/volumes" Nov 01 20:18:08.496979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:08.496933 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:09.313412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.313375 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:09.313675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.313656 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:09.314413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.314392 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:18:09.365209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.365174 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:09.365676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.365603 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:09.366368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.366321 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:18:09.797814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.797779 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf"] Nov 01 20:18:09.803221 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.803201 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:09.855062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.855030 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:18:09.855281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.855128 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:18:09.855281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.855184 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:18:09.855398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.855317 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:18:09.857586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.857557 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf"] Nov 01 20:18:09.901711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.901676 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:09.901884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.901868 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:09.902191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:09.901925 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsg9d\" (UniqueName: \"kubernetes.io/projected/b6b17db6-5e97-4290-a91e-649391f27ce4-kube-api-access-nsg9d\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.003556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.003520 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.003714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.003586 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsg9d\" (UniqueName: \"kubernetes.io/projected/b6b17db6-5e97-4290-a91e-649391f27ce4-kube-api-access-nsg9d\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.003714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.003642 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.004375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.004347 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.006362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.006317 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.060506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.060436 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsg9d\" (UniqueName: \"kubernetes.io/projected/b6b17db6-5e97-4290-a91e-649391f27ce4-kube-api-access-nsg9d\") pod \"maas-api-revoke-sub-8c8fd0ce-xh9zf\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.117674 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.117639 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:10.498809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.498777 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf"] Nov 01 20:18:10.499679 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:18:10.499648 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b17db6_5e97_4290_a91e_649391f27ce4.slice/crio-d7cf6800a946a7a882d3a382df91a174f254d39487ca750b7cb404b2057227b2 WatchSource:0}: Error finding container d7cf6800a946a7a882d3a382df91a174f254d39487ca750b7cb404b2057227b2: Status 404 returned error can't find the container with id d7cf6800a946a7a882d3a382df91a174f254d39487ca750b7cb404b2057227b2 Nov 01 20:18:10.580323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:10.580286 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:11.297605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:11.297561 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" event={"ID":"b6b17db6-5e97-4290-a91e-649391f27ce4","Type":"ContainerStarted","Data":"906720b9e7610aa4bfebf6da1d70f1dc22841a99a312d4faa646a5b92db0b539"} Nov 01 20:18:11.297816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:11.297614 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" event={"ID":"b6b17db6-5e97-4290-a91e-649391f27ce4","Type":"ContainerStarted","Data":"d7cf6800a946a7a882d3a382df91a174f254d39487ca750b7cb404b2057227b2"} Nov 01 20:18:11.411411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:11.411353 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" podStartSLOduration=2.411319747 podStartE2EDuration="2.411319747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:11.409593136 +0000 UTC m=+899.083852393" watchObservedRunningTime="2026-11-01 20:18:11.411319747 +0000 UTC m=+899.085579002" Nov 01 20:18:12.306534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:12.306499 5633 generic.go:356] "Generic (PLEG): container finished" podID="b6b17db6-5e97-4290-a91e-649391f27ce4" containerID="906720b9e7610aa4bfebf6da1d70f1dc22841a99a312d4faa646a5b92db0b539" exitCode=0 Nov 01 20:18:12.306959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:12.306594 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" event={"ID":"b6b17db6-5e97-4290-a91e-649391f27ce4","Type":"ContainerDied","Data":"906720b9e7610aa4bfebf6da1d70f1dc22841a99a312d4faa646a5b92db0b539"} Nov 01 20:18:13.476804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.476774 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:13.557088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.557055 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-openshift-ca\") pod \"b6b17db6-5e97-4290-a91e-649391f27ce4\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " Nov 01 20:18:13.557294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.557136 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-xks-ca\") pod \"b6b17db6-5e97-4290-a91e-649391f27ce4\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " Nov 01 20:18:13.557294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.557173 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b17db6-5e97-4290-a91e-649391f27ce4-kube-api-access-nsg9d\" (UniqueName: \"kubernetes.io/projected/b6b17db6-5e97-4290-a91e-649391f27ce4-kube-api-access-nsg9d\") pod \"b6b17db6-5e97-4290-a91e-649391f27ce4\" (UID: \"b6b17db6-5e97-4290-a91e-649391f27ce4\") " Nov 01 20:18:13.557601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.557570 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-openshift-ca" pod "b6b17db6-5e97-4290-a91e-649391f27ce4" (UID: "b6b17db6-5e97-4290-a91e-649391f27ce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:13.558037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.557953 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:13.559777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.559742 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-xks-ca" pod "b6b17db6-5e97-4290-a91e-649391f27ce4" (UID: "b6b17db6-5e97-4290-a91e-649391f27ce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:18:13.559890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.559867 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b17db6-5e97-4290-a91e-649391f27ce4-kube-api-access-nsg9d" pod "b6b17db6-5e97-4290-a91e-649391f27ce4" (UID: "b6b17db6-5e97-4290-a91e-649391f27ce4"). InnerVolumeSpecName "kube-api-access-nsg9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:13.659097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.659059 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b17db6-5e97-4290-a91e-649391f27ce4-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:13.659097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:13.659092 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsg9d\" (UniqueName: \"kubernetes.io/projected/b6b17db6-5e97-4290-a91e-649391f27ce4-kube-api-access-nsg9d\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:14.322810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:14.322780 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" Nov 01 20:18:14.322810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:14.322803 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf" event={"ID":"b6b17db6-5e97-4290-a91e-649391f27ce4","Type":"ContainerDied","Data":"d7cf6800a946a7a882d3a382df91a174f254d39487ca750b7cb404b2057227b2"} Nov 01 20:18:14.323062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:14.322849 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7cf6800a946a7a882d3a382df91a174f254d39487ca750b7cb404b2057227b2" Nov 01 20:18:15.282149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:15.282114 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:16.157749 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:18:16.157715 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:18:19.422288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:19.422249 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:21.172470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.172371 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:21.312823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.312785 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:21.499256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.499221 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw"] Nov 01 20:18:21.548627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.548593 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-381df3-n47bw"] Nov 01 20:18:21.793013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.792917 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k"] Nov 01 20:18:21.793709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.793644 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" containerID="cri-o://51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5" gracePeriod=30 Nov 01 20:18:21.802435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.802400 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:21.926478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:21.926248 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:22.387026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:22.386992 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:22.468207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:22.468168 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e234c5-6fcb-48f9-847c-1aa828c392c3" path="/var/lib/kubelet/pods/69e234c5-6fcb-48f9-847c-1aa828c392c3/volumes" Nov 01 20:18:24.105503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:24.105464 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:26.288161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:26.288118 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:27.070950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.070888 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:18:27.155166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155131 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-kube-api-access-997fw\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-kube-api-access-997fw\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155168 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istiod-ca-cert\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155247 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-podinfo\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155291 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-certs\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155315 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-token\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-token\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155354 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-data\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155429 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-envoy\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155579 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-credential-socket\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155616 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-socket\") pod \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\" (UID: \"fad5c1c6-a938-4cd0-98f9-9efb5158aa64\") " Nov 01 20:18:27.155710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155612 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-certs" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:27.156358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155742 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istiod-ca-cert" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:27.156358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155729 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-data" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:27.156358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155825 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-socket" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:27.156358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.155937 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-credential-socket" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:27.156608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.156499 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.156608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.156521 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.156608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.156546 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.156608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.156559 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.156608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.156591 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.157751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.157721 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:27.158102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.158078 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-envoy" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:27.158201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.158182 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-podinfo" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:18:27.158275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.158237 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-token" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:27.158275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.158259 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-kube-api-access-997fw" pod "fad5c1c6-a938-4cd0-98f9-9efb5158aa64" (UID: "fad5c1c6-a938-4cd0-98f9-9efb5158aa64"). InnerVolumeSpecName "kube-api-access-997fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:27.257706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.257675 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997fw\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-kube-api-access-997fw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.257706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.257703 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.257706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.257713 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.257960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.257722 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad5c1c6-a938-4cd0-98f9-9efb5158aa64-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:27.417683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.417573 5633 generic.go:356] "Generic (PLEG): container finished" podID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerID="51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5" exitCode=0 Nov 01 20:18:27.417683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.417646 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" Nov 01 20:18:27.417683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.417644 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" event={"ID":"fad5c1c6-a938-4cd0-98f9-9efb5158aa64","Type":"ContainerDied","Data":"51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5"} Nov 01 20:18:27.418251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.417747 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k" event={"ID":"fad5c1c6-a938-4cd0-98f9-9efb5158aa64","Type":"ContainerDied","Data":"3a3af0f57579cecdeda3f1045efacf79dd22b8164bd6ba20bf9ab0447373f053"} Nov 01 20:18:27.418251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.417778 5633 scope.go:122] "RemoveContainer" containerID="51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5" Nov 01 20:18:27.434937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.434907 5633 scope.go:122] "RemoveContainer" containerID="a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b" Nov 01 20:18:27.437678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.437653 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b"] Nov 01 20:18:27.438256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438244 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" Nov 01 20:18:27.438300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438258 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" Nov 01 20:18:27.438300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438275 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" Nov 01 20:18:27.438300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438281 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" Nov 01 20:18:27.438300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438288 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b6b17db6-5e97-4290-a91e-649391f27ce4" containerName="revoke-keys" Nov 01 20:18:27.438300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438293 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b17db6-5e97-4290-a91e-649391f27ce4" containerName="revoke-keys" Nov 01 20:18:27.438492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438414 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" Nov 01 20:18:27.438492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438425 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" containerName="istio-proxy" Nov 01 20:18:27.438492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.438431 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b17db6-5e97-4290-a91e-649391f27ce4" containerName="revoke-keys" Nov 01 20:18:27.443385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.443361 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.445632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.445615 5633 scope.go:122] "RemoveContainer" containerID="51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5" Nov 01 20:18:27.445948 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:18:27.445929 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5\": container with ID starting with 51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5 not found: ID does not exist" containerID="51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5" Nov 01 20:18:27.446044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.445954 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5"} err="failed to get container status \"51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5\": rpc error: code = NotFound desc = could not find container \"51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5\": container with ID starting with 51882f7ce25846065c0bd3f7213155c49258f1b17e6db60253cc303dd7d154f5 not found: ID does not exist" Nov 01 20:18:27.446044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.445977 5633 scope.go:122] "RemoveContainer" containerID="a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b" Nov 01 20:18:27.446432 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:18:27.446405 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b\": container with ID starting with a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b not found: ID does not exist" containerID="a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b" Nov 01 20:18:27.446540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.446436 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b"} err="failed to get container status \"a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b\": rpc error: code = NotFound desc = could not find container \"a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b\": container with ID starting with a089dd952b6babbff03f6167eba5b56aa8c5f2fbdd0dca2c057c15a6bd50d61b not found: ID does not exist" Nov 01 20:18:27.494686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.494653 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b"] Nov 01 20:18:27.497659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.497636 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"distinct-2-w3-1b1134-kserve-self-signed-certs\"" Nov 01 20:18:27.514574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.514532 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:27.560981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.560943 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-model-cache\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.561182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.561081 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ncj\" (UniqueName: \"kubernetes.io/projected/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-kube-api-access-z4ncj\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.561182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.561131 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-home\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.561370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.561343 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tmp-dir\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.561436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.561394 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-dshm\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.561499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.561449 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tls-certs\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.585061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.585023 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k"] Nov 01 20:18:27.593019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.592978 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-381df3-openshift-default-5ddbccd677-dsh2k"] Nov 01 20:18:27.662377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662345 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tmp-dir\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.662578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662386 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-dshm\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.662578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662420 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tls-certs\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.662578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662521 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-model-cache\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.662760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662641 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ncj\" (UniqueName: \"kubernetes.io/projected/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-kube-api-access-z4ncj\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.662760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662706 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-home\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.662875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662787 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tmp-dir\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.662931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.662872 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-model-cache\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.663040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.663019 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-home\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.664840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.664818 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-dshm\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.665160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.665142 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tls-certs\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.719261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.719226 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ncj\" (UniqueName: \"kubernetes.io/projected/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-kube-api-access-z4ncj\") pod \"distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:27.757470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:27.757430 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:28.161418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.161389 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b"] Nov 01 20:18:28.162089 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:18:28.162062 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab5cb13b_c0f3_4b36_b1c2_eaec509c7f45.slice/crio-a667a216a57a154095157f5cb75e71c46ae34c4c84ac358ff52b34184e87f7fc WatchSource:0}: Error finding container a667a216a57a154095157f5cb75e71c46ae34c4c84ac358ff52b34184e87f7fc: Status 404 returned error can't find the container with id a667a216a57a154095157f5cb75e71c46ae34c4c84ac358ff52b34184e87f7fc Nov 01 20:18:28.428560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.428444 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" event={"ID":"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45","Type":"ContainerStarted","Data":"051f35b43cd3bed88b42dd5311ad6d73e9726b11e80b8cbbe464e66aee942c45"} Nov 01 20:18:28.428560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.428491 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" event={"ID":"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45","Type":"ContainerStarted","Data":"a667a216a57a154095157f5cb75e71c46ae34c4c84ac358ff52b34184e87f7fc"} Nov 01 20:18:28.431147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.431126 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s"] Nov 01 20:18:28.435201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.435177 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.469993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.469955 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad5c1c6-a938-4cd0-98f9-9efb5158aa64" path="/var/lib/kubelet/pods/fad5c1c6-a938-4cd0-98f9-9efb5158aa64/volumes" Nov 01 20:18:28.472876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.472850 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.473048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.473028 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.473167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.473146 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rwk8\" (UniqueName: \"kubernetes.io/projected/908634c7-1ee8-4dfb-8906-7c7687875aeb-kube-api-access-5rwk8\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.487599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.487567 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:18:28.487800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.487672 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:18:28.487866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.487838 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:18:28.487928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.487915 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:18:28.490936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.490910 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s"] Nov 01 20:18:28.545806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.545747 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podStartSLOduration=1.5457310340000001 podStartE2EDuration="1.545731034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:28.542180895 +0000 UTC m=+916.216440151" watchObservedRunningTime="2026-11-01 20:18:28.545731034 +0000 UTC m=+916.219990288" Nov 01 20:18:28.574260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.574227 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwk8\" (UniqueName: \"kubernetes.io/projected/908634c7-1ee8-4dfb-8906-7c7687875aeb-kube-api-access-5rwk8\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.574466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.574374 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.574466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.574453 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.575069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.575044 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.576945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.576924 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.630480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.630450 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwk8\" (UniqueName: \"kubernetes.io/projected/908634c7-1ee8-4dfb-8906-7c7687875aeb-kube-api-access-5rwk8\") pod \"maas-api-revoke-sub-4cae6732-5mp9s\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.749731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.749694 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:28.757915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:28.757880 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:29.132923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:29.132896 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s"] Nov 01 20:18:29.133370 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:18:29.133310 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908634c7_1ee8_4dfb_8906_7c7687875aeb.slice/crio-2c0786e50db7700792c7a0eb284c9ed9d64e6825c0a66e2bfb8b2afb2abf665e WatchSource:0}: Error finding container 2c0786e50db7700792c7a0eb284c9ed9d64e6825c0a66e2bfb8b2afb2abf665e: Status 404 returned error can't find the container with id 2c0786e50db7700792c7a0eb284c9ed9d64e6825c0a66e2bfb8b2afb2abf665e Nov 01 20:18:29.436356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:29.436284 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" event={"ID":"908634c7-1ee8-4dfb-8906-7c7687875aeb","Type":"ContainerStarted","Data":"a8456b8155f1c0354512790cda1928b25175396319983a02e0a3c485c809d223"} Nov 01 20:18:29.436904 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:29.436381 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" event={"ID":"908634c7-1ee8-4dfb-8906-7c7687875aeb","Type":"ContainerStarted","Data":"2c0786e50db7700792c7a0eb284c9ed9d64e6825c0a66e2bfb8b2afb2abf665e"} Nov 01 20:18:29.552083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:29.552033 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" podStartSLOduration=1.552016566 podStartE2EDuration="1.552016566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:29.548004586 +0000 UTC m=+917.222263841" watchObservedRunningTime="2026-11-01 20:18:29.552016566 +0000 UTC m=+917.226275820" Nov 01 20:18:30.445358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:30.445242 5633 generic.go:356] "Generic (PLEG): container finished" podID="908634c7-1ee8-4dfb-8906-7c7687875aeb" containerID="a8456b8155f1c0354512790cda1928b25175396319983a02e0a3c485c809d223" exitCode=0 Nov 01 20:18:30.445749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:30.445360 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" event={"ID":"908634c7-1ee8-4dfb-8906-7c7687875aeb","Type":"ContainerDied","Data":"a8456b8155f1c0354512790cda1928b25175396319983a02e0a3c485c809d223"} Nov 01 20:18:30.984204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:30.984159 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:31.341518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.341434 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:18:31.635208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.635181 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:31.721259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.721223 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908634c7-1ee8-4dfb-8906-7c7687875aeb-kube-api-access-5rwk8\" (UniqueName: \"kubernetes.io/projected/908634c7-1ee8-4dfb-8906-7c7687875aeb-kube-api-access-5rwk8\") pod \"908634c7-1ee8-4dfb-8906-7c7687875aeb\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " Nov 01 20:18:31.721463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.721367 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-xks-ca\") pod \"908634c7-1ee8-4dfb-8906-7c7687875aeb\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " Nov 01 20:18:31.721463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.721429 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-openshift-ca\") pod \"908634c7-1ee8-4dfb-8906-7c7687875aeb\" (UID: \"908634c7-1ee8-4dfb-8906-7c7687875aeb\") " Nov 01 20:18:31.722011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.721959 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-openshift-ca" pod "908634c7-1ee8-4dfb-8906-7c7687875aeb" (UID: "908634c7-1ee8-4dfb-8906-7c7687875aeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:31.724365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.723909 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908634c7-1ee8-4dfb-8906-7c7687875aeb-kube-api-access-5rwk8" pod "908634c7-1ee8-4dfb-8906-7c7687875aeb" (UID: "908634c7-1ee8-4dfb-8906-7c7687875aeb"). InnerVolumeSpecName "kube-api-access-5rwk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:31.724365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.724117 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-xks-ca" pod "908634c7-1ee8-4dfb-8906-7c7687875aeb" (UID: "908634c7-1ee8-4dfb-8906-7c7687875aeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:18:31.823624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.823588 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rwk8\" (UniqueName: \"kubernetes.io/projected/908634c7-1ee8-4dfb-8906-7c7687875aeb-kube-api-access-5rwk8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:31.823624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.823621 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:31.823624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:31.823633 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908634c7-1ee8-4dfb-8906-7c7687875aeb-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:32.470800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:32.470764 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" Nov 01 20:18:32.473715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:32.473681 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s" event={"ID":"908634c7-1ee8-4dfb-8906-7c7687875aeb","Type":"ContainerDied","Data":"2c0786e50db7700792c7a0eb284c9ed9d64e6825c0a66e2bfb8b2afb2abf665e"} Nov 01 20:18:32.473906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:32.473890 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0786e50db7700792c7a0eb284c9ed9d64e6825c0a66e2bfb8b2afb2abf665e" Nov 01 20:18:33.419430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:33.419392 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:33.612214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:33.612178 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:33.745137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:33.745074 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:35.487380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:35.487291 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:37.440928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:37.440889 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:37.758072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:37.758032 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:37.771109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:37.771083 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:38.557813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:38.557774 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:39.162880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:39.162845 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:42.283920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.283887 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:42.421294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.421256 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7"] Nov 01 20:18:42.421531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.421503 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" containerID="cri-o://315dc25c30eca1646a5f9fca79f0928acee11e7519955e21c583457a67dfeebf" gracePeriod=30 Nov 01 20:18:42.471301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.471271 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw"] Nov 01 20:18:42.471532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.471501 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" containerID="cri-o://3623057e348f7ba626b215c9786b0198a411f1b16c927e79ed1a687c8a9cb76f" gracePeriod=30 Nov 01 20:18:42.474940 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.474914 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8"] Nov 01 20:18:42.475192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.475157 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" containerID="cri-o://21ba42efdfa18e508592940a8f92ea029ca415bad4b01986bfcb5e55310e9928" gracePeriod=30 Nov 01 20:18:42.520796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.520765 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq"] Nov 01 20:18:42.520981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:42.520953 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" containerID="cri-o://be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a" gracePeriod=30 Nov 01 20:18:44.388823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.388778 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q"] Nov 01 20:18:44.390637 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.390605 5633 cpu_manager.go:414] "containerMap: removing container" podUID="908634c7-1ee8-4dfb-8906-7c7687875aeb" containerName="revoke-keys" Nov 01 20:18:44.390637 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.390633 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908634c7-1ee8-4dfb-8906-7c7687875aeb" containerName="revoke-keys" Nov 01 20:18:44.390818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.390758 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="908634c7-1ee8-4dfb-8906-7c7687875aeb" containerName="revoke-keys" Nov 01 20:18:44.394487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.394466 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.440684 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.440649 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q"] Nov 01 20:18:44.485821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.485790 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:18:44.485990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.485840 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:18:44.485990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.485840 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:18:44.485990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.485955 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:18:44.494060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.494020 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.494249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.494071 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.494249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.494186 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vgj\" (UniqueName: \"kubernetes.io/projected/2f89f6a1-1435-45c9-9692-538d897f0f16-kube-api-access-l2vgj\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.595314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.595263 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.595528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.595323 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.595528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.595403 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vgj\" (UniqueName: \"kubernetes.io/projected/2f89f6a1-1435-45c9-9692-538d897f0f16-kube-api-access-l2vgj\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.596140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.596109 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.597928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.597902 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.652640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.652549 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vgj\" (UniqueName: \"kubernetes.io/projected/2f89f6a1-1435-45c9-9692-538d897f0f16-kube-api-access-l2vgj\") pod \"maas-api-revoke-sub-a16967dc-vmt9q\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:44.709503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:44.709464 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:45.095846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:45.095819 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q"] Nov 01 20:18:45.096825 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:18:45.096799 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f89f6a1_1435_45c9_9692_538d897f0f16.slice/crio-75edbb2d174f19ebfd94a3ffd281162229cebd0ae1ffb68e884a910ed84922d2 WatchSource:0}: Error finding container 75edbb2d174f19ebfd94a3ffd281162229cebd0ae1ffb68e884a910ed84922d2: Status 404 returned error can't find the container with id 75edbb2d174f19ebfd94a3ffd281162229cebd0ae1ffb68e884a910ed84922d2 Nov 01 20:18:45.597843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:45.597810 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" event={"ID":"2f89f6a1-1435-45c9-9692-538d897f0f16","Type":"ContainerStarted","Data":"4b455e7405e61b4410f3319179e0b2625da437b3ab8c2f341e2c86e003417d73"} Nov 01 20:18:45.598202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:45.597862 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" event={"ID":"2f89f6a1-1435-45c9-9692-538d897f0f16","Type":"ContainerStarted","Data":"75edbb2d174f19ebfd94a3ffd281162229cebd0ae1ffb68e884a910ed84922d2"} Nov 01 20:18:45.714065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:45.714011 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" podStartSLOduration=1.713993806 podStartE2EDuration="1.713993806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:45.710663136 +0000 UTC m=+933.384922393" watchObservedRunningTime="2026-11-01 20:18:45.713993806 +0000 UTC m=+933.388253079" Nov 01 20:18:46.158012 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:18:46.157975 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:18:46.607121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:46.607084 5633 generic.go:356] "Generic (PLEG): container finished" podID="2f89f6a1-1435-45c9-9692-538d897f0f16" containerID="4b455e7405e61b4410f3319179e0b2625da437b3ab8c2f341e2c86e003417d73" exitCode=0 Nov 01 20:18:46.607560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:46.607172 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" event={"ID":"2f89f6a1-1435-45c9-9692-538d897f0f16","Type":"ContainerDied","Data":"4b455e7405e61b4410f3319179e0b2625da437b3ab8c2f341e2c86e003417d73"} Nov 01 20:18:47.001409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.001368 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:47.620871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.620845 5633 generic.go:356] "Generic (PLEG): container finished" podID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerID="315dc25c30eca1646a5f9fca79f0928acee11e7519955e21c583457a67dfeebf" exitCode=0 Nov 01 20:18:47.621264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.620921 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" event={"ID":"7fe2f318-5931-41e9-93c2-612b264b8ed5","Type":"ContainerDied","Data":"315dc25c30eca1646a5f9fca79f0928acee11e7519955e21c583457a67dfeebf"} Nov 01 20:18:47.622944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.622915 5633 generic.go:356] "Generic (PLEG): container finished" podID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerID="21ba42efdfa18e508592940a8f92ea029ca415bad4b01986bfcb5e55310e9928" exitCode=0 Nov 01 20:18:47.623062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.622983 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" event={"ID":"c113b0da-2c36-4f5d-8557-390ba1c244b5","Type":"ContainerDied","Data":"21ba42efdfa18e508592940a8f92ea029ca415bad4b01986bfcb5e55310e9928"} Nov 01 20:18:47.624691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.624668 5633 generic.go:356] "Generic (PLEG): container finished" podID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerID="3623057e348f7ba626b215c9786b0198a411f1b16c927e79ed1a687c8a9cb76f" exitCode=0 Nov 01 20:18:47.624952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.624934 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" event={"ID":"5fbfb2e8-b991-464c-8145-6a07f6250f6f","Type":"ContainerDied","Data":"3623057e348f7ba626b215c9786b0198a411f1b16c927e79ed1a687c8a9cb76f"} Nov 01 20:18:47.781008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.780971 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:18:47.859188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859093 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-credential-socket\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859183 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-token\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-token\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859241 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-envoy\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859291 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-podinfo\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859360 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe2f318-5931-41e9-93c2-612b264b8ed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2f318-5931-41e9-93c2-612b264b8ed5-istiod-ca-cert\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859394 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-data\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859559 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-kube-api-access-787wh\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-kube-api-access-787wh\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859696 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-certs\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.859822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.859794 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-socket\") pod \"7fe2f318-5931-41e9-93c2-612b264b8ed5\" (UID: \"7fe2f318-5931-41e9-93c2-612b264b8ed5\") " Nov 01 20:18:47.863111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.860775 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-socket" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.863111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.861114 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe2f318-5931-41e9-93c2-612b264b8ed5-istiod-ca-cert" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:47.863111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.861377 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-data" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.864189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.864147 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-credential-socket" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.865501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.865457 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-certs" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.886565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.886443 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:18:47.893682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.893625 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-podinfo" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:18:47.894247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.894204 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-envoy" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.894583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.894550 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-kube-api-access-787wh" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "kube-api-access-787wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:47.900511 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.900476 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-token" pod "7fe2f318-5931-41e9-93c2-612b264b8ed5" (UID: "7fe2f318-5931-41e9-93c2-612b264b8ed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961317 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-data\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961385 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-kube-api-access-xbrtt\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-kube-api-access-xbrtt\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961495 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-socket\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961527 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-envoy\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961570 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-certs\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961615 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c113b0da-2c36-4f5d-8557-390ba1c244b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c113b0da-2c36-4f5d-8557-390ba1c244b5-istiod-ca-cert\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961666 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-podinfo\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961731 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-token\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-token\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.961771 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-credential-socket\") pod \"c113b0da-2c36-4f5d-8557-390ba1c244b5\" (UID: \"c113b0da-2c36-4f5d-8557-390ba1c244b5\") " Nov 01 20:18:47.962461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962280 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-certs" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962508 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962528 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962541 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962554 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962566 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962578 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962590 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962602 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2f318-5931-41e9-93c2-612b264b8ed5-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962615 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe2f318-5931-41e9-93c2-612b264b8ed5-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962627 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-787wh\" (UniqueName: \"kubernetes.io/projected/7fe2f318-5931-41e9-93c2-612b264b8ed5-kube-api-access-787wh\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:47.963076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.962757 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-data" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.964039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.964015 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-socket" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.966057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.964672 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c113b0da-2c36-4f5d-8557-390ba1c244b5-istiod-ca-cert" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:47.966057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.964695 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-kube-api-access-xbrtt" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "kube-api-access-xbrtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:47.966248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.966094 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-credential-socket" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.966969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.966943 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-podinfo" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:18:47.967081 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.967030 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-token" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:47.971928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.971601 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-envoy" pod "c113b0da-2c36-4f5d-8557-390ba1c244b5" (UID: "c113b0da-2c36-4f5d-8557-390ba1c244b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:47.985275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.985241 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:18:47.990490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.990422 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:18:47.992763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:47.992738 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:48.063290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063247 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-socket\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063304 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-data\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063360 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-envoy\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063397 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-credential-socket\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063432 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-token\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-token\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063467 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-podinfo\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063521 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-kube-api-access-pwkkb\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-kube-api-access-pwkkb\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063568 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-certs\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063611 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1710eb6f-7d43-4a4f-9d6a-799369f80228-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1710eb6f-7d43-4a4f-9d6a-799369f80228-istiod-ca-cert\") pod \"1710eb6f-7d43-4a4f-9d6a-799369f80228\" (UID: \"1710eb6f-7d43-4a4f-9d6a-799369f80228\") " Nov 01 20:18:48.063860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063667 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-socket" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.063860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.063685 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-data" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.064223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064195 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-credential-socket" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.064319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064206 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064231 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1710eb6f-7d43-4a4f-9d6a-799369f80228-istiod-ca-cert" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:48.064319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064256 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064275 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c113b0da-2c36-4f5d-8557-390ba1c244b5-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064290 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064303 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064288 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-certs" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.064678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064317 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064447 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064462 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064515 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c113b0da-2c36-4f5d-8557-390ba1c244b5-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.064678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.064538 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrtt\" (UniqueName: \"kubernetes.io/projected/c113b0da-2c36-4f5d-8557-390ba1c244b5-kube-api-access-xbrtt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.066619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.066562 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-podinfo" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:18:48.066619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.066593 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-kube-api-access-pwkkb" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "kube-api-access-pwkkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:48.067044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.067021 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-envoy" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.067140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.067059 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-token" pod "1710eb6f-7d43-4a4f-9d6a-799369f80228" (UID: "1710eb6f-7d43-4a4f-9d6a-799369f80228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167110 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-data\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167166 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istiod-ca-cert\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167228 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-socket\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167283 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-kube-api-access-svkhr\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-kube-api-access-svkhr\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167320 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-envoy\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167426 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-openshift-ca\") pod \"2f89f6a1-1435-45c9-9692-538d897f0f16\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167511 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-podinfo\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167542 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-token\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167667 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-xks-ca\") pod \"2f89f6a1-1435-45c9-9692-538d897f0f16\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167797 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-credential-socket\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167830 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f89f6a1-1435-45c9-9692-538d897f0f16-kube-api-access-l2vgj\" (UniqueName: \"kubernetes.io/projected/2f89f6a1-1435-45c9-9692-538d897f0f16-kube-api-access-l2vgj\") pod \"2f89f6a1-1435-45c9-9692-538d897f0f16\" (UID: \"2f89f6a1-1435-45c9-9692-538d897f0f16\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.167862 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-certs\") pod \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\" (UID: \"5fbfb2e8-b991-464c-8145-6a07f6250f6f\") " Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.168570 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1710eb6f-7d43-4a4f-9d6a-799369f80228-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.168594 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.168612 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.168625 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.168637 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1710eb6f-7d43-4a4f-9d6a-799369f80228-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.168650 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwkkb\" (UniqueName: \"kubernetes.io/projected/1710eb6f-7d43-4a4f-9d6a-799369f80228-kube-api-access-pwkkb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.168665 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1710eb6f-7d43-4a4f-9d6a-799369f80228-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.169255 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-openshift-ca" pod "2f89f6a1-1435-45c9-9692-538d897f0f16" (UID: "2f89f6a1-1435-45c9-9692-538d897f0f16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.169566 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-data" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.169874 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istiod-ca-cert" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.170113 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-socket" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.170786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.170466 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-certs" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.175346 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.175284 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-credential-socket" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.178403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.178317 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f89f6a1-1435-45c9-9692-538d897f0f16-kube-api-access-l2vgj" pod "2f89f6a1-1435-45c9-9692-538d897f0f16" (UID: "2f89f6a1-1435-45c9-9692-538d897f0f16"). InnerVolumeSpecName "kube-api-access-l2vgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:48.182224 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.182195 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-xks-ca" pod "2f89f6a1-1435-45c9-9692-538d897f0f16" (UID: "2f89f6a1-1435-45c9-9692-538d897f0f16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:18:48.187057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.187028 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-podinfo" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:18:48.187189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.187139 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-kube-api-access-svkhr" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "kube-api-access-svkhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:48.188730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.188705 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-envoy" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:18:48.190385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.190166 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-token" pod "5fbfb2e8-b991-464c-8145-6a07f6250f6f" (UID: "5fbfb2e8-b991-464c-8145-6a07f6250f6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270074 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svkhr\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-kube-api-access-svkhr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270117 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270133 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270147 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270161 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270174 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f89f6a1-1435-45c9-9692-538d897f0f16-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270187 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270200 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2vgj\" (UniqueName: \"kubernetes.io/projected/2f89f6a1-1435-45c9-9692-538d897f0f16-kube-api-access-l2vgj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270214 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270229 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270242 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfb2e8-b991-464c-8145-6a07f6250f6f-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.270297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.270256 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfb2e8-b991-464c-8145-6a07f6250f6f-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:48.690430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.690390 5633 generic.go:356] "Generic (PLEG): container finished" podID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerID="be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a" exitCode=0 Nov 01 20:18:48.691144 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.691112 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" event={"ID":"1710eb6f-7d43-4a4f-9d6a-799369f80228","Type":"ContainerDied","Data":"be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a"} Nov 01 20:18:48.691383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.691359 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" event={"ID":"1710eb6f-7d43-4a4f-9d6a-799369f80228","Type":"ContainerDied","Data":"e909e837792cba52b831ae884db1b0afd53e1622ddd14af62444f9c118e25904"} Nov 01 20:18:48.691520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.691507 5633 scope.go:122] "RemoveContainer" containerID="be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a" Nov 01 20:18:48.691790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.691773 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq" Nov 01 20:18:48.697304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.697281 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" event={"ID":"2f89f6a1-1435-45c9-9692-538d897f0f16","Type":"ContainerDied","Data":"75edbb2d174f19ebfd94a3ffd281162229cebd0ae1ffb68e884a910ed84922d2"} Nov 01 20:18:48.697622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.697603 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75edbb2d174f19ebfd94a3ffd281162229cebd0ae1ffb68e884a910ed84922d2" Nov 01 20:18:48.697847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.697832 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q" Nov 01 20:18:48.706103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.706071 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" event={"ID":"7fe2f318-5931-41e9-93c2-612b264b8ed5","Type":"ContainerDied","Data":"5246963d83b51a5fd31979b7f3a5ffc1f925e4f1c62c63cf8e0b38cc61753fce"} Nov 01 20:18:48.706421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.706355 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7" Nov 01 20:18:48.715629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.715041 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" event={"ID":"c113b0da-2c36-4f5d-8557-390ba1c244b5","Type":"ContainerDied","Data":"3aeaca5a0492930d4d7fe5d61eb3941942b817dfc02813f7f473da4c98fd488b"} Nov 01 20:18:48.715948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.715923 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8" Nov 01 20:18:48.721717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.721683 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" event={"ID":"5fbfb2e8-b991-464c-8145-6a07f6250f6f","Type":"ContainerDied","Data":"3f46eea56a084594d62e7bb6de3fc10a7f56c64f7f3914657ec5b01ee4bd9773"} Nov 01 20:18:48.721983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.721964 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw" Nov 01 20:18:48.736808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.736783 5633 scope.go:122] "RemoveContainer" containerID="be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a" Nov 01 20:18:48.737500 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:18:48.737458 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a\": container with ID starting with be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a not found: ID does not exist" containerID="be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a" Nov 01 20:18:48.737629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.737500 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a"} err="failed to get container status \"be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a\": rpc error: code = NotFound desc = could not find container \"be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a\": container with ID starting with be6838c4a8c572aeae04e5086a63e4ceb639633f78df8b20f1c224b016a9ee1a not found: ID does not exist" Nov 01 20:18:48.737629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.737530 5633 scope.go:122] "RemoveContainer" containerID="315dc25c30eca1646a5f9fca79f0928acee11e7519955e21c583457a67dfeebf" Nov 01 20:18:48.756711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.756683 5633 scope.go:122] "RemoveContainer" containerID="21ba42efdfa18e508592940a8f92ea029ca415bad4b01986bfcb5e55310e9928" Nov 01 20:18:48.774300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.774259 5633 scope.go:122] "RemoveContainer" containerID="3623057e348f7ba626b215c9786b0198a411f1b16c927e79ed1a687c8a9cb76f" Nov 01 20:18:48.875424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.874297 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7"] Nov 01 20:18:48.880357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:48.880261 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lwr7"] Nov 01 20:18:49.020804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:49.020730 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw"] Nov 01 20:18:49.074359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:49.050905 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wd9hw"] Nov 01 20:18:49.207180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:49.207043 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq"] Nov 01 20:18:49.220457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:49.220400 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bp6cq"] Nov 01 20:18:49.307969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:49.307901 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8"] Nov 01 20:18:49.326498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:49.326458 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rphz8"] Nov 01 20:18:50.270359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:50.267443 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:18:50.518376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:50.508032 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" path="/var/lib/kubelet/pods/1710eb6f-7d43-4a4f-9d6a-799369f80228/volumes" Nov 01 20:18:50.518376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:50.508815 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" path="/var/lib/kubelet/pods/5fbfb2e8-b991-464c-8145-6a07f6250f6f/volumes" Nov 01 20:18:50.522449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:50.521573 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" path="/var/lib/kubelet/pods/7fe2f318-5931-41e9-93c2-612b264b8ed5/volumes" Nov 01 20:18:50.522449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:50.522309 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" path="/var/lib/kubelet/pods/c113b0da-2c36-4f5d-8557-390ba1c244b5/volumes" Nov 01 20:18:50.556371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:50.550437 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 01 20:18:50.575478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:50.575417 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:18:51.268482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:51.268429 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:18:51.269304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:51.269277 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:18:51.669455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:51.666988 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 01 20:18:51.669455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:51.667769 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:51.739496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:51.739460 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:18:52.142914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.122481 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:18:52.458707 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.458615 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:18:52.458955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.458925 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:18:52.569411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.564486 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:18:52.656305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.656271 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5"] Nov 01 20:18:52.657469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.657377 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" Nov 01 20:18:52.657645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.657609 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" Nov 01 20:18:52.657813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.657788 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" Nov 01 20:18:52.657928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.657915 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" Nov 01 20:18:52.658024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658013 5633 cpu_manager.go:414] "containerMap: removing container" podUID="2f89f6a1-1435-45c9-9692-538d897f0f16" containerName="revoke-keys" Nov 01 20:18:52.658112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658103 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f89f6a1-1435-45c9-9692-538d897f0f16" containerName="revoke-keys" Nov 01 20:18:52.658186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658177 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" Nov 01 20:18:52.658253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658244 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" Nov 01 20:18:52.658355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658345 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" Nov 01 20:18:52.658437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658428 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" Nov 01 20:18:52.658711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658701 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbfb2e8-b991-464c-8145-6a07f6250f6f" containerName="istio-proxy" Nov 01 20:18:52.658794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658786 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1710eb6f-7d43-4a4f-9d6a-799369f80228" containerName="istio-proxy" Nov 01 20:18:52.658863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658855 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe2f318-5931-41e9-93c2-612b264b8ed5" containerName="istio-proxy" Nov 01 20:18:52.658931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658923 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f89f6a1-1435-45c9-9692-538d897f0f16" containerName="revoke-keys" Nov 01 20:18:52.659002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.658994 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c113b0da-2c36-4f5d-8557-390ba1c244b5" containerName="istio-proxy" Nov 01 20:18:52.671610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.671578 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.767359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.758926 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7d5118-openshift-default-dockercfg-2rtxd\"" Nov 01 20:18:52.828966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.825454 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5"] Nov 01 20:18:52.927606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.927491 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.927913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.927887 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.928137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.928118 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bz6\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-kube-api-access-h2bz6\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.928402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.928383 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.928644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.928625 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.928829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.928812 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.928976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.928962 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.930651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.930507 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.930651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.930573 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:52.965887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:52.965837 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031633 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031694 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031715 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031746 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031784 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031849 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031867 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031922 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bz6\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-kube-api-access-h2bz6\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.031983 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.032733 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.032897 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.033117 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.034367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.033763 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.041596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.035028 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.043078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.043052 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.053797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.050399 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.093779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.093717 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bz6\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-kube-api-access-h2bz6\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.095763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.095738 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.139577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.139382 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:53.576843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.576811 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5"] Nov 01 20:18:53.580910 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:18:53.580875 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cd5a75_e0c0_479d_aaf0_5a424f40d60c.slice/crio-4db7fd4a1a190ec3f9037686cfcec8ba65c19d8fb364af041edc77d5a034cf9a WatchSource:0}: Error finding container 4db7fd4a1a190ec3f9037686cfcec8ba65c19d8fb364af041edc77d5a034cf9a: Status 404 returned error can't find the container with id 4db7fd4a1a190ec3f9037686cfcec8ba65c19d8fb364af041edc77d5a034cf9a Nov 01 20:18:53.583991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.583833 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:53.583991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.583934 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:53.584172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:53.584005 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:18:54.199414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.199372 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" event={"ID":"31cd5a75-e0c0-479d-aaf0-5a424f40d60c","Type":"ContainerStarted","Data":"e3e4ad74c860591ffb1152d7b5766662a3e7424a6b46e43166926592d0917da5"} Nov 01 20:18:54.199933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.199443 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" event={"ID":"31cd5a75-e0c0-479d-aaf0-5a424f40d60c","Type":"ContainerStarted","Data":"4db7fd4a1a190ec3f9037686cfcec8ba65c19d8fb364af041edc77d5a034cf9a"} Nov 01 20:18:54.242383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.242126 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt"] Nov 01 20:18:54.279794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.279752 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.325874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.325841 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt"] Nov 01 20:18:54.333699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.333658 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:18:54.333884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.333674 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:18:54.333884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.333771 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:18:54.333884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.333841 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:18:54.358554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.358496 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podStartSLOduration=2.3584821480000002 podStartE2EDuration="2.358482148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:54.356506458 +0000 UTC m=+942.030765714" watchObservedRunningTime="2026-11-01 20:18:54.358482148 +0000 UTC m=+942.032741402" Nov 01 20:18:54.362767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.362730 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.362956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.362834 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cqv\" (UniqueName: \"kubernetes.io/projected/1d50f956-e566-4dd6-8f67-7856a0365066-kube-api-access-58cqv\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.362956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.362913 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.463757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.463721 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cqv\" (UniqueName: \"kubernetes.io/projected/1d50f956-e566-4dd6-8f67-7856a0365066-kube-api-access-58cqv\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.463968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.463789 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.463968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.463903 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.464557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.464525 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.467477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.467447 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.523139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.523105 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cqv\" (UniqueName: \"kubernetes.io/projected/1d50f956-e566-4dd6-8f67-7856a0365066-kube-api-access-58cqv\") pod \"maas-api-revoke-sub-3ab19483-mzjbt\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:54.633882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:54.633846 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:55.121405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:55.115375 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt"] Nov 01 20:18:55.142732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:55.141703 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:18:55.142732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:55.141867 5633 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:18:55.142732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:55.141915 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 20:18:55.236872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:55.236431 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" event={"ID":"1d50f956-e566-4dd6-8f67-7856a0365066","Type":"ContainerStarted","Data":"ca78c08508da84d741e85c97a66bb63870565e0c40852e707d5fb58ae95c9d22"} Nov 01 20:18:56.140116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:56.139949 5633 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:18:56.140116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:56.140005 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 20:18:56.248438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:56.248396 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" event={"ID":"1d50f956-e566-4dd6-8f67-7856a0365066","Type":"ContainerStarted","Data":"31d36dac24597a2a530565760df2986d2fdde791dd264a9dd4991ad51d2e2ff0"} Nov 01 20:18:56.367826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:56.367757 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" podStartSLOduration=2.367735989 podStartE2EDuration="2.367735989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:56.362571973 +0000 UTC m=+944.036831228" watchObservedRunningTime="2026-11-01 20:18:56.367735989 +0000 UTC m=+944.041995244" Nov 01 20:18:57.140352 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:57.140307 5633 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:18:57.140545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:57.140381 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 20:18:57.258738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:57.258700 5633 generic.go:356] "Generic (PLEG): container finished" podID="1d50f956-e566-4dd6-8f67-7856a0365066" containerID="31d36dac24597a2a530565760df2986d2fdde791dd264a9dd4991ad51d2e2ff0" exitCode=0 Nov 01 20:18:57.259109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:57.258790 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" event={"ID":"1d50f956-e566-4dd6-8f67-7856a0365066","Type":"ContainerDied","Data":"31d36dac24597a2a530565760df2986d2fdde791dd264a9dd4991ad51d2e2ff0"} Nov 01 20:18:57.999356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:57.994921 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:18:58.140388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.140352 5633 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:18:58.140677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.140654 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 20:18:58.626294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.626262 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:18:58.777440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.776275 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-openshift-ca\") pod \"1d50f956-e566-4dd6-8f67-7856a0365066\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " Nov 01 20:18:58.777440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.776404 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-xks-ca\") pod \"1d50f956-e566-4dd6-8f67-7856a0365066\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " Nov 01 20:18:58.777440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.776474 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d50f956-e566-4dd6-8f67-7856a0365066-kube-api-access-58cqv\" (UniqueName: \"kubernetes.io/projected/1d50f956-e566-4dd6-8f67-7856a0365066-kube-api-access-58cqv\") pod \"1d50f956-e566-4dd6-8f67-7856a0365066\" (UID: \"1d50f956-e566-4dd6-8f67-7856a0365066\") " Nov 01 20:18:58.794906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.794867 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-openshift-ca" pod "1d50f956-e566-4dd6-8f67-7856a0365066" (UID: "1d50f956-e566-4dd6-8f67-7856a0365066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:58.826395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.826239 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-xks-ca" pod "1d50f956-e566-4dd6-8f67-7856a0365066" (UID: "1d50f956-e566-4dd6-8f67-7856a0365066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:18:58.826737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.826714 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d50f956-e566-4dd6-8f67-7856a0365066-kube-api-access-58cqv" pod "1d50f956-e566-4dd6-8f67-7856a0365066" (UID: "1d50f956-e566-4dd6-8f67-7856a0365066"). InnerVolumeSpecName "kube-api-access-58cqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:58.905363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.895845 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cqv\" (UniqueName: \"kubernetes.io/projected/1d50f956-e566-4dd6-8f67-7856a0365066-kube-api-access-58cqv\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:58.905363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.895901 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:58.905363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:58.895919 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d50f956-e566-4dd6-8f67-7856a0365066-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:18:59.146430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:59.142787 5633 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:18:59.146430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:59.142859 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 20:18:59.291388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:59.289003 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" event={"ID":"1d50f956-e566-4dd6-8f67-7856a0365066","Type":"ContainerDied","Data":"ca78c08508da84d741e85c97a66bb63870565e0c40852e707d5fb58ae95c9d22"} Nov 01 20:18:59.291388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:59.289088 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca78c08508da84d741e85c97a66bb63870565e0c40852e707d5fb58ae95c9d22" Nov 01 20:18:59.291388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:18:59.289204 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt" Nov 01 20:19:00.140492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.140458 5633 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 20:19:00.140919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.140512 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 20:19:00.207991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.207956 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z"] Nov 01 20:19:00.208587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.208570 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1d50f956-e566-4dd6-8f67-7856a0365066" containerName="revoke-keys" Nov 01 20:19:00.208587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.208586 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d50f956-e566-4dd6-8f67-7856a0365066" containerName="revoke-keys" Nov 01 20:19:00.208785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.208772 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d50f956-e566-4dd6-8f67-7856a0365066" containerName="revoke-keys" Nov 01 20:19:00.214032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.214006 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.266477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.266442 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z"] Nov 01 20:19:00.316443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.315800 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:19:00.316651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.316479 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:19:00.316651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.316593 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:19:00.316798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.316705 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:19:00.338969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.338927 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.339170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.339000 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8d8\" (UniqueName: \"kubernetes.io/projected/2aeae86d-b416-4c82-ab20-ed96588c1c68-kube-api-access-vm8d8\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.339266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.339235 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.440627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.440511 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.440627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.440608 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8d8\" (UniqueName: \"kubernetes.io/projected/2aeae86d-b416-4c82-ab20-ed96588c1c68-kube-api-access-vm8d8\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.440867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.440714 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.441665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.441637 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.443465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.443434 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.505191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.505160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8d8\" (UniqueName: \"kubernetes.io/projected/2aeae86d-b416-4c82-ab20-ed96588c1c68-kube-api-access-vm8d8\") pod \"maas-api-revoke-sub-c2f044b9-gp64z\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.535365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.533552 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:00.967602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:00.967578 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z"] Nov 01 20:19:00.968706 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:00.968683 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aeae86d_b416_4c82_ab20_ed96588c1c68.slice/crio-d0b271bd8a4679d229675ff5e1058a01b78e153c12f2ac591c6ad61f92471827 WatchSource:0}: Error finding container d0b271bd8a4679d229675ff5e1058a01b78e153c12f2ac591c6ad61f92471827: Status 404 returned error can't find the container with id d0b271bd8a4679d229675ff5e1058a01b78e153c12f2ac591c6ad61f92471827 Nov 01 20:19:01.144409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:01.144379 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:19:01.144826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:01.144805 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:19:01.145569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:01.145550 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:19:01.345100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:01.345057 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" event={"ID":"2aeae86d-b416-4c82-ab20-ed96588c1c68","Type":"ContainerStarted","Data":"5459302c943a880da8770f93aa1224a5c85626f4b9e74696f461427aa8f7a198"} Nov 01 20:19:01.345288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:01.345120 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" event={"ID":"2aeae86d-b416-4c82-ab20-ed96588c1c68","Type":"ContainerStarted","Data":"d0b271bd8a4679d229675ff5e1058a01b78e153c12f2ac591c6ad61f92471827"} Nov 01 20:19:01.489671 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:01.489618 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" podStartSLOduration=1.48960356 podStartE2EDuration="1.48960356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:01.487586837 +0000 UTC m=+949.161846092" watchObservedRunningTime="2026-11-01 20:19:01.48960356 +0000 UTC m=+949.163862847" Nov 01 20:19:02.353598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:02.353564 5633 generic.go:356] "Generic (PLEG): container finished" podID="2aeae86d-b416-4c82-ab20-ed96588c1c68" containerID="5459302c943a880da8770f93aa1224a5c85626f4b9e74696f461427aa8f7a198" exitCode=0 Nov 01 20:19:02.353999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:02.353653 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" event={"ID":"2aeae86d-b416-4c82-ab20-ed96588c1c68","Type":"ContainerDied","Data":"5459302c943a880da8770f93aa1224a5c85626f4b9e74696f461427aa8f7a198"} Nov 01 20:19:02.782174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:02.782133 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:03.512051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.512024 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:03.606827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.606786 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-xks-ca\") pod \"2aeae86d-b416-4c82-ab20-ed96588c1c68\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " Nov 01 20:19:03.607036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.606842 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-openshift-ca\") pod \"2aeae86d-b416-4c82-ab20-ed96588c1c68\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " Nov 01 20:19:03.607036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.606892 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aeae86d-b416-4c82-ab20-ed96588c1c68-kube-api-access-vm8d8\" (UniqueName: \"kubernetes.io/projected/2aeae86d-b416-4c82-ab20-ed96588c1c68-kube-api-access-vm8d8\") pod \"2aeae86d-b416-4c82-ab20-ed96588c1c68\" (UID: \"2aeae86d-b416-4c82-ab20-ed96588c1c68\") " Nov 01 20:19:03.607568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.607267 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-openshift-ca" pod "2aeae86d-b416-4c82-ab20-ed96588c1c68" (UID: "2aeae86d-b416-4c82-ab20-ed96588c1c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:03.607568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.607527 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:03.609150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.609128 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-xks-ca" pod "2aeae86d-b416-4c82-ab20-ed96588c1c68" (UID: "2aeae86d-b416-4c82-ab20-ed96588c1c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:03.609284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.609264 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aeae86d-b416-4c82-ab20-ed96588c1c68-kube-api-access-vm8d8" pod "2aeae86d-b416-4c82-ab20-ed96588c1c68" (UID: "2aeae86d-b416-4c82-ab20-ed96588c1c68"). InnerVolumeSpecName "kube-api-access-vm8d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:03.709194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.709161 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8d8\" (UniqueName: \"kubernetes.io/projected/2aeae86d-b416-4c82-ab20-ed96588c1c68-kube-api-access-vm8d8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:03.709194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:03.709193 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aeae86d-b416-4c82-ab20-ed96588c1c68-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:04.370909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:04.370876 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" Nov 01 20:19:04.371091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:04.370911 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z" event={"ID":"2aeae86d-b416-4c82-ab20-ed96588c1c68","Type":"ContainerDied","Data":"d0b271bd8a4679d229675ff5e1058a01b78e153c12f2ac591c6ad61f92471827"} Nov 01 20:19:04.371091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:04.370956 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0b271bd8a4679d229675ff5e1058a01b78e153c12f2ac591c6ad61f92471827" Nov 01 20:19:05.550809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:05.550770 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:19:08.741481 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:08.741428 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:19:09.212446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:09.210273 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:09.365843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:09.365454 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:19:09.365843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:09.365543 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:19:09.479570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:09.479445 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:19:10.545834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.545793 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg"] Nov 01 20:19:10.546422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.546408 5633 cpu_manager.go:414] "containerMap: removing container" podUID="2aeae86d-b416-4c82-ab20-ed96588c1c68" containerName="revoke-keys" Nov 01 20:19:10.546472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.546423 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aeae86d-b416-4c82-ab20-ed96588c1c68" containerName="revoke-keys" Nov 01 20:19:10.546539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.546530 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aeae86d-b416-4c82-ab20-ed96588c1c68" containerName="revoke-keys" Nov 01 20:19:10.550371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.550319 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.606487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.606453 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg"] Nov 01 20:19:10.621375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.620819 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.621375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.620894 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.621375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.620926 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvct\" (UniqueName: \"kubernetes.io/projected/14856dd8-8d3a-492d-8752-d73dec42d4e4-kube-api-access-fmvct\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.651027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.650988 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:19:10.651286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.651066 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:19:10.651286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.651071 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:19:10.651488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.651087 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:19:10.722112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.722077 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.722300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.722133 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.722300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.722152 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvct\" (UniqueName: \"kubernetes.io/projected/14856dd8-8d3a-492d-8752-d73dec42d4e4-kube-api-access-fmvct\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.722939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.722917 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.724931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.724903 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.779679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.779647 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvct\" (UniqueName: \"kubernetes.io/projected/14856dd8-8d3a-492d-8752-d73dec42d4e4-kube-api-access-fmvct\") pod \"maas-api-revoke-sub-048ffa5e-mlqbg\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:10.864572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:10.864471 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:11.242783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:11.242574 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg"] Nov 01 20:19:11.242962 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:11.242939 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14856dd8_8d3a_492d_8752_d73dec42d4e4.slice/crio-468063a2ee63e27ae4c53b8f214af4a4cd19033d8c0af7154fe2771ea59f079e WatchSource:0}: Error finding container 468063a2ee63e27ae4c53b8f214af4a4cd19033d8c0af7154fe2771ea59f079e: Status 404 returned error can't find the container with id 468063a2ee63e27ae4c53b8f214af4a4cd19033d8c0af7154fe2771ea59f079e Nov 01 20:19:11.574868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:11.574834 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" event={"ID":"14856dd8-8d3a-492d-8752-d73dec42d4e4","Type":"ContainerStarted","Data":"f79f5b5f9cfafa7d59ab943374d4bc70692bef3c4a70a3154b14e8a3443d884e"} Nov 01 20:19:11.575243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:11.574883 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" event={"ID":"14856dd8-8d3a-492d-8752-d73dec42d4e4","Type":"ContainerStarted","Data":"468063a2ee63e27ae4c53b8f214af4a4cd19033d8c0af7154fe2771ea59f079e"} Nov 01 20:19:11.690536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:11.690480 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" podStartSLOduration=1.690461381 podStartE2EDuration="1.690461381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:11.687006261 +0000 UTC m=+959.361265529" watchObservedRunningTime="2026-11-01 20:19:11.690461381 +0000 UTC m=+959.364720635" Nov 01 20:19:12.584576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:12.584531 5633 generic.go:356] "Generic (PLEG): container finished" podID="14856dd8-8d3a-492d-8752-d73dec42d4e4" containerID="f79f5b5f9cfafa7d59ab943374d4bc70692bef3c4a70a3154b14e8a3443d884e" exitCode=0 Nov 01 20:19:12.584979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:12.584667 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" event={"ID":"14856dd8-8d3a-492d-8752-d73dec42d4e4","Type":"ContainerDied","Data":"f79f5b5f9cfafa7d59ab943374d4bc70692bef3c4a70a3154b14e8a3443d884e"} Nov 01 20:19:12.880306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:12.880225 5633 scope.go:122] "RemoveContainer" containerID="e3fbfd23b2cc065893d22b5fbe6d8bbd4dc9054f42bed31f0aa9ed05492ae949" Nov 01 20:19:13.752657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.752625 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:13.859622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.859534 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-xks-ca\") pod \"14856dd8-8d3a-492d-8752-d73dec42d4e4\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " Nov 01 20:19:13.859622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.859615 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14856dd8-8d3a-492d-8752-d73dec42d4e4-kube-api-access-fmvct\" (UniqueName: \"kubernetes.io/projected/14856dd8-8d3a-492d-8752-d73dec42d4e4-kube-api-access-fmvct\") pod \"14856dd8-8d3a-492d-8752-d73dec42d4e4\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " Nov 01 20:19:13.859861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.859833 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-openshift-ca\") pod \"14856dd8-8d3a-492d-8752-d73dec42d4e4\" (UID: \"14856dd8-8d3a-492d-8752-d73dec42d4e4\") " Nov 01 20:19:13.860187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.860147 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-openshift-ca" pod "14856dd8-8d3a-492d-8752-d73dec42d4e4" (UID: "14856dd8-8d3a-492d-8752-d73dec42d4e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:13.860638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.860615 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:13.861873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.861842 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-xks-ca" pod "14856dd8-8d3a-492d-8752-d73dec42d4e4" (UID: "14856dd8-8d3a-492d-8752-d73dec42d4e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:13.861989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.861930 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14856dd8-8d3a-492d-8752-d73dec42d4e4-kube-api-access-fmvct" pod "14856dd8-8d3a-492d-8752-d73dec42d4e4" (UID: "14856dd8-8d3a-492d-8752-d73dec42d4e4"). InnerVolumeSpecName "kube-api-access-fmvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:13.962037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.962000 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14856dd8-8d3a-492d-8752-d73dec42d4e4-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:13.962037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:13.962029 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmvct\" (UniqueName: \"kubernetes.io/projected/14856dd8-8d3a-492d-8752-d73dec42d4e4-kube-api-access-fmvct\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:14.600826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:14.600794 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" Nov 01 20:19:14.600826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:14.600805 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg" event={"ID":"14856dd8-8d3a-492d-8752-d73dec42d4e4","Type":"ContainerDied","Data":"468063a2ee63e27ae4c53b8f214af4a4cd19033d8c0af7154fe2771ea59f079e"} Nov 01 20:19:14.601032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:14.600850 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="468063a2ee63e27ae4c53b8f214af4a4cd19033d8c0af7154fe2771ea59f079e" Nov 01 20:19:15.089279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:15.089248 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:16.160918 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:19:16.160863 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:19:19.106407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.106372 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v"] Nov 01 20:19:19.107153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.107131 5633 cpu_manager.go:414] "containerMap: removing container" podUID="14856dd8-8d3a-492d-8752-d73dec42d4e4" containerName="revoke-keys" Nov 01 20:19:19.107254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.107155 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14856dd8-8d3a-492d-8752-d73dec42d4e4" containerName="revoke-keys" Nov 01 20:19:19.107381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.107367 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="14856dd8-8d3a-492d-8752-d73dec42d4e4" containerName="revoke-keys" Nov 01 20:19:19.111095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.111074 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.165824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.165790 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v"] Nov 01 20:19:19.210983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.210951 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:19:19.212253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.212228 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:19:19.212431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.212274 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:19:19.212431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.212278 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:19:19.224959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.224932 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbn6\" (UniqueName: \"kubernetes.io/projected/f42cc7b1-6701-43d1-a9ce-81e634d32e43-kube-api-access-6bbn6\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.225089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.225052 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.225146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.225132 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.326879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.326841 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbn6\" (UniqueName: \"kubernetes.io/projected/f42cc7b1-6701-43d1-a9ce-81e634d32e43-kube-api-access-6bbn6\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.327080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.326939 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.327080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.327011 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.327730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.327704 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.329835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.329806 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.383046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.382968 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbn6\" (UniqueName: \"kubernetes.io/projected/f42cc7b1-6701-43d1-a9ce-81e634d32e43-kube-api-access-6bbn6\") pod \"maas-api-revoke-sub-0bde0792-sjp8v\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.424679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.424634 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:19.578349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.578292 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:19.748977 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.748936 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:19.802455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:19.802424 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v"] Nov 01 20:19:19.802787 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:19.802764 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42cc7b1_6701_43d1_a9ce_81e634d32e43.slice/crio-88cedb617c2dc9f50a9c2415f6596864914c08a20204c35179cd8a17ea2b7a36 WatchSource:0}: Error finding container 88cedb617c2dc9f50a9c2415f6596864914c08a20204c35179cd8a17ea2b7a36: Status 404 returned error can't find the container with id 88cedb617c2dc9f50a9c2415f6596864914c08a20204c35179cd8a17ea2b7a36 Nov 01 20:19:20.101294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:20.101262 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:20.718976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:20.718940 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" event={"ID":"f42cc7b1-6701-43d1-a9ce-81e634d32e43","Type":"ContainerStarted","Data":"8eb69e5438f6fe3a6e541d620b7d3660b8a5cf1c561d672998007b7191a87e70"} Nov 01 20:19:20.719438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:20.719008 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" event={"ID":"f42cc7b1-6701-43d1-a9ce-81e634d32e43","Type":"ContainerStarted","Data":"88cedb617c2dc9f50a9c2415f6596864914c08a20204c35179cd8a17ea2b7a36"} Nov 01 20:19:20.834594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:20.834538 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" podStartSLOduration=1.834521558 podStartE2EDuration="1.834521558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:20.833316892 +0000 UTC m=+968.507576148" watchObservedRunningTime="2026-11-01 20:19:20.834521558 +0000 UTC m=+968.508780812" Nov 01 20:19:21.149137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:21.149027 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:21.728540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:21.728496 5633 generic.go:356] "Generic (PLEG): container finished" podID="f42cc7b1-6701-43d1-a9ce-81e634d32e43" containerID="8eb69e5438f6fe3a6e541d620b7d3660b8a5cf1c561d672998007b7191a87e70" exitCode=0 Nov 01 20:19:21.728948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:21.728562 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" event={"ID":"f42cc7b1-6701-43d1-a9ce-81e634d32e43","Type":"ContainerDied","Data":"8eb69e5438f6fe3a6e541d620b7d3660b8a5cf1c561d672998007b7191a87e70"} Nov 01 20:19:21.774659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:21.774625 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:22.717864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.717825 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:22.897656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.897629 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:22.969811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.969721 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42cc7b1-6701-43d1-a9ce-81e634d32e43-kube-api-access-6bbn6\" (UniqueName: \"kubernetes.io/projected/f42cc7b1-6701-43d1-a9ce-81e634d32e43-kube-api-access-6bbn6\") pod \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " Nov 01 20:19:22.969811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.969797 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-openshift-ca\") pod \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " Nov 01 20:19:22.970044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.969887 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-xks-ca\") pod \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\" (UID: \"f42cc7b1-6701-43d1-a9ce-81e634d32e43\") " Nov 01 20:19:22.970236 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.970211 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-openshift-ca" pod "f42cc7b1-6701-43d1-a9ce-81e634d32e43" (UID: "f42cc7b1-6701-43d1-a9ce-81e634d32e43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:22.970499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.970478 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:22.972211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.972187 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-xks-ca" pod "f42cc7b1-6701-43d1-a9ce-81e634d32e43" (UID: "f42cc7b1-6701-43d1-a9ce-81e634d32e43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:22.972442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:22.972426 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42cc7b1-6701-43d1-a9ce-81e634d32e43-kube-api-access-6bbn6" pod "f42cc7b1-6701-43d1-a9ce-81e634d32e43" (UID: "f42cc7b1-6701-43d1-a9ce-81e634d32e43"). InnerVolumeSpecName "kube-api-access-6bbn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:23.071673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.071641 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbn6\" (UniqueName: \"kubernetes.io/projected/f42cc7b1-6701-43d1-a9ce-81e634d32e43-kube-api-access-6bbn6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:23.071673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.071671 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc7b1-6701-43d1-a9ce-81e634d32e43-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:23.436516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.436422 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:23.744952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.744925 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" Nov 01 20:19:23.744952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.744945 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v" event={"ID":"f42cc7b1-6701-43d1-a9ce-81e634d32e43","Type":"ContainerDied","Data":"88cedb617c2dc9f50a9c2415f6596864914c08a20204c35179cd8a17ea2b7a36"} Nov 01 20:19:23.745158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.744998 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88cedb617c2dc9f50a9c2415f6596864914c08a20204c35179cd8a17ea2b7a36" Nov 01 20:19:23.950575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.950532 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b"] Nov 01 20:19:23.951462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.951354 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f42cc7b1-6701-43d1-a9ce-81e634d32e43" containerName="revoke-keys" Nov 01 20:19:23.951462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.951374 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42cc7b1-6701-43d1-a9ce-81e634d32e43" containerName="revoke-keys" Nov 01 20:19:23.951630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.951559 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42cc7b1-6701-43d1-a9ce-81e634d32e43" containerName="revoke-keys" Nov 01 20:19:23.955384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.955362 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.983881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.983846 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-certs\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.983914 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-data\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.983982 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-token\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.984033 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-envoy\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.984063 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-socket\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.984159 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-credential-socket\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.984260 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-podinfo\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.984374 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ede63f-d465-4d69-bda7-3800a6f7d614-istiod-ca-cert\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:23.984520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:23.984417 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gps49\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-kube-api-access-gps49\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.034078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.033986 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:24.053497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.053464 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b"] Nov 01 20:19:24.067218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.067192 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-99d504-openshift-default-dockercfg-55qzx\"" Nov 01 20:19:24.085634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.085598 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-certs\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.085847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.085662 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-data\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.085847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.085698 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-token\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.085847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.085792 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-envoy\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.085847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.085825 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-socket\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086016 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-credential-socket\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086190 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-podinfo\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086371 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-certs\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086256 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-socket\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086495 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-credential-socket\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086614 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-data\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086756 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ede63f-d465-4d69-bda7-3800a6f7d614-istiod-ca-cert\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.086808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.086814 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps49\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-kube-api-access-gps49\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.087710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.087684 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ede63f-d465-4d69-bda7-3800a6f7d614-istiod-ca-cert\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.088594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.088566 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-envoy\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.088910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.088889 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-podinfo\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.152489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.149647 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gps49\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-kube-api-access-gps49\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.152489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.150839 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-token\") pod \"e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.269993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.269959 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:24.379568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.376979 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:24.723505 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:24.723469 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ede63f_d465_4d69_bda7_3800a6f7d614.slice/crio-e6c490d62d4ea87dce6748435e85e5aeea33d64c56b5a9ed05436c091abe0625 WatchSource:0}: Error finding container e6c490d62d4ea87dce6748435e85e5aeea33d64c56b5a9ed05436c091abe0625: Status 404 returned error can't find the container with id e6c490d62d4ea87dce6748435e85e5aeea33d64c56b5a9ed05436c091abe0625 Nov 01 20:19:24.723955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.723930 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b"] Nov 01 20:19:24.726299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.726267 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:19:24.726409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.726384 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:19:24.726534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.726428 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:19:24.753261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:24.753226 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" event={"ID":"61ede63f-d465-4d69-bda7-3800a6f7d614","Type":"ContainerStarted","Data":"e6c490d62d4ea87dce6748435e85e5aeea33d64c56b5a9ed05436c091abe0625"} Nov 01 20:19:25.352847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:25.352794 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:25.762415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:25.762379 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" event={"ID":"61ede63f-d465-4d69-bda7-3800a6f7d614","Type":"ContainerStarted","Data":"49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6"} Nov 01 20:19:25.933692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:25.933617 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" podStartSLOduration=2.9335970270000002 podStartE2EDuration="2.933597027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:25.927075395 +0000 UTC m=+973.601334651" watchObservedRunningTime="2026-11-01 20:19:25.933597027 +0000 UTC m=+973.607856283" Nov 01 20:19:26.271088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:26.271045 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:26.276831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:26.276803 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:26.769656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:26.769622 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:26.770705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:26.770681 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:19:28.054692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.054651 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv"] Nov 01 20:19:28.059089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.059060 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.123915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.123882 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv"] Nov 01 20:19:28.139862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.139823 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.140025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.139964 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwl9\" (UniqueName: \"kubernetes.io/projected/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-kube-api-access-8fwl9\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.140175 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.140158 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.168682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.168643 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:19:28.168682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.168681 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:19:28.168902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.168726 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:19:28.168902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.168799 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:19:28.241924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.241886 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.242102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.242024 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.242102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.242075 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwl9\" (UniqueName: \"kubernetes.io/projected/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-kube-api-access-8fwl9\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.242520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.242502 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.244620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.244596 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.299992 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.299950 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwl9\" (UniqueName: \"kubernetes.io/projected/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-kube-api-access-8fwl9\") pod \"maas-api-revoke-sub-57294611-x26pv\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.374861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.374777 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:28.696419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.696168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:28.752387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.752305 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv"] Nov 01 20:19:28.752520 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:28.752492 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3252f4e_a48c_4fa4_a55b_1e9ba4580db6.slice/crio-698b0d812fc94db75a9ebf7bcbadd06bc7ccd48697a1f24b2a304629f87b405c WatchSource:0}: Error finding container 698b0d812fc94db75a9ebf7bcbadd06bc7ccd48697a1f24b2a304629f87b405c: Status 404 returned error can't find the container with id 698b0d812fc94db75a9ebf7bcbadd06bc7ccd48697a1f24b2a304629f87b405c Nov 01 20:19:28.787936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.787900 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" event={"ID":"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6","Type":"ContainerStarted","Data":"698b0d812fc94db75a9ebf7bcbadd06bc7ccd48697a1f24b2a304629f87b405c"} Nov 01 20:19:28.891299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:28.891244 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:29.798390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:29.798355 5633 generic.go:356] "Generic (PLEG): container finished" podID="d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" containerID="163c66c68b96f27f7a30cfbc8ddc1e3a5521f43541adcbd7f077ba92a9b861b9" exitCode=0 Nov 01 20:19:29.798806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:29.798402 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" event={"ID":"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6","Type":"ContainerDied","Data":"163c66c68b96f27f7a30cfbc8ddc1e3a5521f43541adcbd7f077ba92a9b861b9"} Nov 01 20:19:29.954639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:29.954602 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs"] Nov 01 20:19:29.959244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:29.959219 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.024513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.024438 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs"] Nov 01 20:19:30.070046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.070006 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192caabb-218b-42fa-9b74-d30328130b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.070260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.070216 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192caabb-218b-42fa-9b74-d30328130b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.070260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.070250 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzxr\" (UniqueName: \"kubernetes.io/projected/192caabb-218b-42fa-9b74-d30328130b18-kube-api-access-nlzxr\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.171974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.171940 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192caabb-218b-42fa-9b74-d30328130b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.172189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.171991 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzxr\" (UniqueName: \"kubernetes.io/projected/192caabb-218b-42fa-9b74-d30328130b18-kube-api-access-nlzxr\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.172189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.172070 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192caabb-218b-42fa-9b74-d30328130b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.172629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.172601 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192caabb-218b-42fa-9b74-d30328130b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.174646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.174623 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192caabb-218b-42fa-9b74-d30328130b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.228860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.228823 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzxr\" (UniqueName: \"kubernetes.io/projected/192caabb-218b-42fa-9b74-d30328130b18-kube-api-access-nlzxr\") pod \"maas-api-revoke-sub-4c2aa8df-f56cs\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.273608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.273570 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:30.327068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.326762 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:19:30.444867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.444828 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:30.683018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.682982 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs"] Nov 01 20:19:30.685118 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:30.685087 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192caabb_218b_42fa_9b74_d30328130b18.slice/crio-b9d880a6ac5b978a0d606ec1b5667d672804d55e55e4a157365f6e345024d79c WatchSource:0}: Error finding container b9d880a6ac5b978a0d606ec1b5667d672804d55e55e4a157365f6e345024d79c: Status 404 returned error can't find the container with id b9d880a6ac5b978a0d606ec1b5667d672804d55e55e4a157365f6e345024d79c Nov 01 20:19:30.829359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:30.828478 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" event={"ID":"192caabb-218b-42fa-9b74-d30328130b18","Type":"ContainerStarted","Data":"b9d880a6ac5b978a0d606ec1b5667d672804d55e55e4a157365f6e345024d79c"} Nov 01 20:19:31.175559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.175202 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:31.333355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.333014 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-kube-api-access-8fwl9\" (UniqueName: \"kubernetes.io/projected/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-kube-api-access-8fwl9\") pod \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " Nov 01 20:19:31.333355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.333143 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-xks-ca\") pod \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " Nov 01 20:19:31.333355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.333200 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-openshift-ca\") pod \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\" (UID: \"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6\") " Nov 01 20:19:31.353358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.334582 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-openshift-ca" pod "d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" (UID: "d3252f4e-a48c-4fa4-a55b-1e9ba4580db6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:31.353358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.351205 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-xks-ca" pod "d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" (UID: "d3252f4e-a48c-4fa4-a55b-1e9ba4580db6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:31.378360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.370750 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-kube-api-access-8fwl9" pod "d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" (UID: "d3252f4e-a48c-4fa4-a55b-1e9ba4580db6"). InnerVolumeSpecName "kube-api-access-8fwl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:31.488359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.460252 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fwl9\" (UniqueName: \"kubernetes.io/projected/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-kube-api-access-8fwl9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:31.488359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.460299 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:31.488359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.460314 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:31.934358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.931109 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" event={"ID":"192caabb-218b-42fa-9b74-d30328130b18","Type":"ContainerStarted","Data":"dae33e3ff354471f68bd06078087aa45d8adf491b652a1e437760920cc1b84a7"} Nov 01 20:19:31.954930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.950984 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" event={"ID":"d3252f4e-a48c-4fa4-a55b-1e9ba4580db6","Type":"ContainerDied","Data":"698b0d812fc94db75a9ebf7bcbadd06bc7ccd48697a1f24b2a304629f87b405c"} Nov 01 20:19:31.954930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.951051 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="698b0d812fc94db75a9ebf7bcbadd06bc7ccd48697a1f24b2a304629f87b405c" Nov 01 20:19:31.954930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:31.951120 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv" Nov 01 20:19:32.112313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:32.081600 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" podStartSLOduration=3.081577501 podStartE2EDuration="3.081577501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:32.079771302 +0000 UTC m=+979.754030558" watchObservedRunningTime="2026-11-01 20:19:32.081577501 +0000 UTC m=+979.755836756" Nov 01 20:19:32.413423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:32.404006 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:32.898538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:32.898491 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:33.262359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:33.242437 5633 generic.go:356] "Generic (PLEG): container finished" podID="192caabb-218b-42fa-9b74-d30328130b18" containerID="dae33e3ff354471f68bd06078087aa45d8adf491b652a1e437760920cc1b84a7" exitCode=0 Nov 01 20:19:33.262359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:33.242572 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" event={"ID":"192caabb-218b-42fa-9b74-d30328130b18","Type":"ContainerDied","Data":"dae33e3ff354471f68bd06078087aa45d8adf491b652a1e437760920cc1b84a7"} Nov 01 20:19:33.793927 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:33.793879 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:19:34.214422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:34.211468 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 01 20:19:34.587624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:34.587521 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:19:35.027372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.026005 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:35.036865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.036161 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/192caabb-218b-42fa-9b74-d30328130b18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192caabb-218b-42fa-9b74-d30328130b18-maas-api-xks-ca\") pod \"192caabb-218b-42fa-9b74-d30328130b18\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " Nov 01 20:19:35.036865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.036237 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192caabb-218b-42fa-9b74-d30328130b18-kube-api-access-nlzxr\" (UniqueName: \"kubernetes.io/projected/192caabb-218b-42fa-9b74-d30328130b18-kube-api-access-nlzxr\") pod \"192caabb-218b-42fa-9b74-d30328130b18\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " Nov 01 20:19:35.036865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.036290 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192caabb-218b-42fa-9b74-d30328130b18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192caabb-218b-42fa-9b74-d30328130b18-maas-api-openshift-ca\") pod \"192caabb-218b-42fa-9b74-d30328130b18\" (UID: \"192caabb-218b-42fa-9b74-d30328130b18\") " Nov 01 20:19:35.037401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.037317 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192caabb-218b-42fa-9b74-d30328130b18-maas-api-openshift-ca" pod "192caabb-218b-42fa-9b74-d30328130b18" (UID: "192caabb-218b-42fa-9b74-d30328130b18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:35.039822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.039781 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192caabb-218b-42fa-9b74-d30328130b18-kube-api-access-nlzxr" pod "192caabb-218b-42fa-9b74-d30328130b18" (UID: "192caabb-218b-42fa-9b74-d30328130b18"). InnerVolumeSpecName "kube-api-access-nlzxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:35.043924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.043880 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192caabb-218b-42fa-9b74-d30328130b18-maas-api-xks-ca" pod "192caabb-218b-42fa-9b74-d30328130b18" (UID: "192caabb-218b-42fa-9b74-d30328130b18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:35.137841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.137810 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/192caabb-218b-42fa-9b74-d30328130b18-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:35.137841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.137840 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzxr\" (UniqueName: \"kubernetes.io/projected/192caabb-218b-42fa-9b74-d30328130b18-kube-api-access-nlzxr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:35.138132 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.137857 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/192caabb-218b-42fa-9b74-d30328130b18-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:35.329587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.329547 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" event={"ID":"192caabb-218b-42fa-9b74-d30328130b18","Type":"ContainerDied","Data":"b9d880a6ac5b978a0d606ec1b5667d672804d55e55e4a157365f6e345024d79c"} Nov 01 20:19:35.329587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.329586 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs" Nov 01 20:19:35.329808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:35.329606 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9d880a6ac5b978a0d606ec1b5667d672804d55e55e4a157365f6e345024d79c" Nov 01 20:19:36.285764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:36.285733 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:38.785024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.784986 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4"] Nov 01 20:19:38.785793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.785772 5633 cpu_manager.go:414] "containerMap: removing container" podUID="192caabb-218b-42fa-9b74-d30328130b18" containerName="revoke-keys" Nov 01 20:19:38.785905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.785796 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192caabb-218b-42fa-9b74-d30328130b18" containerName="revoke-keys" Nov 01 20:19:38.785905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.785830 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" containerName="revoke-keys" Nov 01 20:19:38.785905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.785838 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" containerName="revoke-keys" Nov 01 20:19:38.786060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.785995 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="192caabb-218b-42fa-9b74-d30328130b18" containerName="revoke-keys" Nov 01 20:19:38.786060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.786013 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" containerName="revoke-keys" Nov 01 20:19:38.789817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.789793 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.844524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.844493 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4"] Nov 01 20:19:38.884392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.884358 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ed387-344f-4683-804f-b4a3052ced10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.884618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.884423 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnq7\" (UniqueName: \"kubernetes.io/projected/949ed387-344f-4683-804f-b4a3052ced10-kube-api-access-6bnq7\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.884618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.884590 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ed387-344f-4683-804f-b4a3052ced10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.889614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.889553 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:19:38.889614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.889586 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:19:38.889614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.889607 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:19:38.891251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.891227 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:19:38.986073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.986037 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ed387-344f-4683-804f-b4a3052ced10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.986228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.986168 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ed387-344f-4683-804f-b4a3052ced10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.986228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.986215 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnq7\" (UniqueName: \"kubernetes.io/projected/949ed387-344f-4683-804f-b4a3052ced10-kube-api-access-6bnq7\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.986808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.986783 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ed387-344f-4683-804f-b4a3052ced10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:38.988798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:38.988766 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ed387-344f-4683-804f-b4a3052ced10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:39.042961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:39.042886 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnq7\" (UniqueName: \"kubernetes.io/projected/949ed387-344f-4683-804f-b4a3052ced10-kube-api-access-6bnq7\") pod \"maas-api-revoke-sub-52ba28de-sw4l4\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:39.103570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:39.103533 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:39.499993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:39.499954 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4"] Nov 01 20:19:39.508427 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:39.508397 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949ed387_344f_4683_804f_b4a3052ced10.slice/crio-6da3dd1a289eaf0f177ab8c06d889be40d5bc9387cf56666f7fe979123eaadc1 WatchSource:0}: Error finding container 6da3dd1a289eaf0f177ab8c06d889be40d5bc9387cf56666f7fe979123eaadc1: Status 404 returned error can't find the container with id 6da3dd1a289eaf0f177ab8c06d889be40d5bc9387cf56666f7fe979123eaadc1 Nov 01 20:19:40.390076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:40.390040 5633 generic.go:356] "Generic (PLEG): container finished" podID="949ed387-344f-4683-804f-b4a3052ced10" containerID="c6dee954ae2fcd70a66429802d7baa00e9d3d64d3b06c775dd573743973b9f8a" exitCode=0 Nov 01 20:19:40.390520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:40.390127 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" event={"ID":"949ed387-344f-4683-804f-b4a3052ced10","Type":"ContainerDied","Data":"c6dee954ae2fcd70a66429802d7baa00e9d3d64d3b06c775dd573743973b9f8a"} Nov 01 20:19:40.390520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:40.390193 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" event={"ID":"949ed387-344f-4683-804f-b4a3052ced10","Type":"ContainerStarted","Data":"6da3dd1a289eaf0f177ab8c06d889be40d5bc9387cf56666f7fe979123eaadc1"} Nov 01 20:19:40.994286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:40.994250 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:41.554709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.554654 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:41.622699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.622640 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949ed387-344f-4683-804f-b4a3052ced10-kube-api-access-6bnq7\" (UniqueName: \"kubernetes.io/projected/949ed387-344f-4683-804f-b4a3052ced10-kube-api-access-6bnq7\") pod \"949ed387-344f-4683-804f-b4a3052ced10\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " Nov 01 20:19:41.623082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.622778 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/949ed387-344f-4683-804f-b4a3052ced10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ed387-344f-4683-804f-b4a3052ced10-maas-api-xks-ca\") pod \"949ed387-344f-4683-804f-b4a3052ced10\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " Nov 01 20:19:41.623082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.622992 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949ed387-344f-4683-804f-b4a3052ced10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ed387-344f-4683-804f-b4a3052ced10-maas-api-openshift-ca\") pod \"949ed387-344f-4683-804f-b4a3052ced10\" (UID: \"949ed387-344f-4683-804f-b4a3052ced10\") " Nov 01 20:19:41.623442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.623392 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949ed387-344f-4683-804f-b4a3052ced10-maas-api-openshift-ca" pod "949ed387-344f-4683-804f-b4a3052ced10" (UID: "949ed387-344f-4683-804f-b4a3052ced10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:41.623582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.623567 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949ed387-344f-4683-804f-b4a3052ced10-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:41.625303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.625277 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949ed387-344f-4683-804f-b4a3052ced10-maas-api-xks-ca" pod "949ed387-344f-4683-804f-b4a3052ced10" (UID: "949ed387-344f-4683-804f-b4a3052ced10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:41.625403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.625285 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949ed387-344f-4683-804f-b4a3052ced10-kube-api-access-6bnq7" pod "949ed387-344f-4683-804f-b4a3052ced10" (UID: "949ed387-344f-4683-804f-b4a3052ced10"). InnerVolumeSpecName "kube-api-access-6bnq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:41.725411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.725379 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnq7\" (UniqueName: \"kubernetes.io/projected/949ed387-344f-4683-804f-b4a3052ced10-kube-api-access-6bnq7\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:41.725411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:41.725409 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949ed387-344f-4683-804f-b4a3052ced10-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:42.406531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:42.406495 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" Nov 01 20:19:42.406688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:42.406525 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4" event={"ID":"949ed387-344f-4683-804f-b4a3052ced10","Type":"ContainerDied","Data":"6da3dd1a289eaf0f177ab8c06d889be40d5bc9387cf56666f7fe979123eaadc1"} Nov 01 20:19:42.406688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:42.406578 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da3dd1a289eaf0f177ab8c06d889be40d5bc9387cf56666f7fe979123eaadc1" Nov 01 20:19:45.689517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.689479 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:45.843729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.843683 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4"] Nov 01 20:19:45.844881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.844851 5633 cpu_manager.go:414] "containerMap: removing container" podUID="949ed387-344f-4683-804f-b4a3052ced10" containerName="revoke-keys" Nov 01 20:19:45.845034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.845020 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949ed387-344f-4683-804f-b4a3052ced10" containerName="revoke-keys" Nov 01 20:19:45.845386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.845369 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="949ed387-344f-4683-804f-b4a3052ced10" containerName="revoke-keys" Nov 01 20:19:45.861974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.861941 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:45.917357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.916963 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4"] Nov 01 20:19:45.961393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.961367 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:19:45.961393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.961378 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:19:45.961615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.961423 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:19:45.963839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.963805 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:19:45.980955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.980920 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:45.981107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.981066 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:45.981146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:45.981116 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bdj\" (UniqueName: \"kubernetes.io/projected/8063796d-dbd8-4656-8e0c-dc2896c7229a-kube-api-access-24bdj\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.082066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.082024 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bdj\" (UniqueName: \"kubernetes.io/projected/8063796d-dbd8-4656-8e0c-dc2896c7229a-kube-api-access-24bdj\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.082244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.082164 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.082244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.082231 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.082899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.082878 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.085082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.085054 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.139495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.139459 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bdj\" (UniqueName: \"kubernetes.io/projected/8063796d-dbd8-4656-8e0c-dc2896c7229a-kube-api-access-24bdj\") pod \"maas-api-revoke-sub-755f1583-h6bb4\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.165835 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:19:46.165789 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:19:46.219409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.214635 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:46.912523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:46.912488 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4"] Nov 01 20:19:47.613851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:47.612617 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" event={"ID":"8063796d-dbd8-4656-8e0c-dc2896c7229a","Type":"ContainerStarted","Data":"26fa9da03abc2caa9e808e0b55830e774c3015cd02773f13b4d0f3739759f5d5"} Nov 01 20:19:47.613851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:47.612697 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" event={"ID":"8063796d-dbd8-4656-8e0c-dc2896c7229a","Type":"ContainerStarted","Data":"d986b59c3ac57cdf46fb007d08f064b7dc89d7e261c2ad224fcc99f714d7f52f"} Nov 01 20:19:47.746046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:47.745976 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" podStartSLOduration=2.745954305 podStartE2EDuration="2.745954305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:47.727237076 +0000 UTC m=+995.401496332" watchObservedRunningTime="2026-11-01 20:19:47.745954305 +0000 UTC m=+995.420213563" Nov 01 20:19:48.763834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:48.763262 5633 generic.go:356] "Generic (PLEG): container finished" podID="8063796d-dbd8-4656-8e0c-dc2896c7229a" containerID="26fa9da03abc2caa9e808e0b55830e774c3015cd02773f13b4d0f3739759f5d5" exitCode=0 Nov 01 20:19:48.763834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:48.763449 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" event={"ID":"8063796d-dbd8-4656-8e0c-dc2896c7229a","Type":"ContainerDied","Data":"26fa9da03abc2caa9e808e0b55830e774c3015cd02773f13b4d0f3739759f5d5"} Nov 01 20:19:49.942840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.942809 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:49.986076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.986040 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-openshift-ca\") pod \"8063796d-dbd8-4656-8e0c-dc2896c7229a\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " Nov 01 20:19:49.986240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.986091 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8063796d-dbd8-4656-8e0c-dc2896c7229a-kube-api-access-24bdj\" (UniqueName: \"kubernetes.io/projected/8063796d-dbd8-4656-8e0c-dc2896c7229a-kube-api-access-24bdj\") pod \"8063796d-dbd8-4656-8e0c-dc2896c7229a\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " Nov 01 20:19:49.986240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.986159 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-xks-ca\") pod \"8063796d-dbd8-4656-8e0c-dc2896c7229a\" (UID: \"8063796d-dbd8-4656-8e0c-dc2896c7229a\") " Nov 01 20:19:49.986492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.986463 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-openshift-ca" pod "8063796d-dbd8-4656-8e0c-dc2896c7229a" (UID: "8063796d-dbd8-4656-8e0c-dc2896c7229a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:49.986875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.986768 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:49.988488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.988466 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-xks-ca" pod "8063796d-dbd8-4656-8e0c-dc2896c7229a" (UID: "8063796d-dbd8-4656-8e0c-dc2896c7229a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:49.988601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:49.988512 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8063796d-dbd8-4656-8e0c-dc2896c7229a-kube-api-access-24bdj" pod "8063796d-dbd8-4656-8e0c-dc2896c7229a" (UID: "8063796d-dbd8-4656-8e0c-dc2896c7229a"). InnerVolumeSpecName "kube-api-access-24bdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:50.088563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:50.088466 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bdj\" (UniqueName: \"kubernetes.io/projected/8063796d-dbd8-4656-8e0c-dc2896c7229a-kube-api-access-24bdj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:50.088563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:50.088510 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8063796d-dbd8-4656-8e0c-dc2896c7229a-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:50.686457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:50.686418 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:50.782197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:50.781959 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" event={"ID":"8063796d-dbd8-4656-8e0c-dc2896c7229a","Type":"ContainerDied","Data":"d986b59c3ac57cdf46fb007d08f064b7dc89d7e261c2ad224fcc99f714d7f52f"} Nov 01 20:19:50.782197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:50.782026 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d986b59c3ac57cdf46fb007d08f064b7dc89d7e261c2ad224fcc99f714d7f52f" Nov 01 20:19:50.782197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:50.782100 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4" Nov 01 20:19:54.998211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:54.998177 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd"] Nov 01 20:19:54.998814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:54.998799 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8063796d-dbd8-4656-8e0c-dc2896c7229a" containerName="revoke-keys" Nov 01 20:19:54.998814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:54.998816 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8063796d-dbd8-4656-8e0c-dc2896c7229a" containerName="revoke-keys" Nov 01 20:19:54.998950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:54.998941 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8063796d-dbd8-4656-8e0c-dc2896c7229a" containerName="revoke-keys" Nov 01 20:19:55.003860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.003833 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.058805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.058772 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd"] Nov 01 20:19:55.060130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.060104 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9mf\" (UniqueName: \"kubernetes.io/projected/2eff0a38-18e7-4952-8436-ebe3b1653436-kube-api-access-9f9mf\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.060260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.060165 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.060260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.060223 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.103239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.103202 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:19:55.103239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.103230 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:19:55.103523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.103412 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:19:55.109010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.108987 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:19:55.161594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.161557 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9mf\" (UniqueName: \"kubernetes.io/projected/2eff0a38-18e7-4952-8436-ebe3b1653436-kube-api-access-9f9mf\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.161779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.161618 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.161779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.161649 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.162444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.162411 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.164211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.164180 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.218122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.218093 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9mf\" (UniqueName: \"kubernetes.io/projected/2eff0a38-18e7-4952-8436-ebe3b1653436-kube-api-access-9f9mf\") pod \"maas-api-revoke-sub-c52890b9-t7skd\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.318692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.318602 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:55.415958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.415921 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:55.695188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.695157 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd"] Nov 01 20:19:55.695484 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:55.695461 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eff0a38_18e7_4952_8436_ebe3b1653436.slice/crio-201c8c6023d4179b9875e4c387f9f311ee0bb77a6f7cd898af172c22b7a05df6 WatchSource:0}: Error finding container 201c8c6023d4179b9875e4c387f9f311ee0bb77a6f7cd898af172c22b7a05df6: Status 404 returned error can't find the container with id 201c8c6023d4179b9875e4c387f9f311ee0bb77a6f7cd898af172c22b7a05df6 Nov 01 20:19:55.823172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:55.823134 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" event={"ID":"2eff0a38-18e7-4952-8436-ebe3b1653436","Type":"ContainerStarted","Data":"201c8c6023d4179b9875e4c387f9f311ee0bb77a6f7cd898af172c22b7a05df6"} Nov 01 20:19:56.604827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.604795 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd"] Nov 01 20:19:56.609009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.608985 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.666636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.666564 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd"] Nov 01 20:19:56.682676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.682644 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.682866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.682690 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.682866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.682716 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq98x\" (UniqueName: \"kubernetes.io/projected/540c652b-3449-4fb9-ad3f-6191f42d95f3-kube-api-access-fq98x\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.784484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.784446 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.784678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.784495 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.784678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.784569 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq98x\" (UniqueName: \"kubernetes.io/projected/540c652b-3449-4fb9-ad3f-6191f42d95f3-kube-api-access-fq98x\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.785201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.785177 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.787197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.787171 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.832281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.832248 5633 generic.go:356] "Generic (PLEG): container finished" podID="2eff0a38-18e7-4952-8436-ebe3b1653436" containerID="baa837faa6efa6a6c39e59d07c896bdddb5b7fbfe373bb875920e597b7017940" exitCode=0 Nov 01 20:19:56.832482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.832351 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" event={"ID":"2eff0a38-18e7-4952-8436-ebe3b1653436","Type":"ContainerDied","Data":"baa837faa6efa6a6c39e59d07c896bdddb5b7fbfe373bb875920e597b7017940"} Nov 01 20:19:56.842941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.842877 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq98x\" (UniqueName: \"kubernetes.io/projected/540c652b-3449-4fb9-ad3f-6191f42d95f3-kube-api-access-fq98x\") pod \"maas-api-revoke-sub-b33ac65c-xg9gd\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:56.978989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:56.978959 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:19:57.356295 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:19:57.356267 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540c652b_3449_4fb9_ad3f_6191f42d95f3.slice/crio-eb9f93a95e7c21d7172c8c9f48945735a5778becbbf38b6284c44a6337f70f0a WatchSource:0}: Error finding container eb9f93a95e7c21d7172c8c9f48945735a5778becbbf38b6284c44a6337f70f0a: Status 404 returned error can't find the container with id eb9f93a95e7c21d7172c8c9f48945735a5778becbbf38b6284c44a6337f70f0a Nov 01 20:19:57.356858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:57.356833 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd"] Nov 01 20:19:57.842064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:57.842025 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" event={"ID":"540c652b-3449-4fb9-ad3f-6191f42d95f3","Type":"ContainerStarted","Data":"07a98b8fec983208f50f2fb68e2f5f53452aa447a9ffe1cc910e97f0ecdc5b05"} Nov 01 20:19:57.842604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:57.842090 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" event={"ID":"540c652b-3449-4fb9-ad3f-6191f42d95f3","Type":"ContainerStarted","Data":"eb9f93a95e7c21d7172c8c9f48945735a5778becbbf38b6284c44a6337f70f0a"} Nov 01 20:19:58.013808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.013780 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:58.071557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.071399 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:58.101539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.101494 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eff0a38-18e7-4952-8436-ebe3b1653436-kube-api-access-9f9mf\" (UniqueName: \"kubernetes.io/projected/2eff0a38-18e7-4952-8436-ebe3b1653436-kube-api-access-9f9mf\") pod \"2eff0a38-18e7-4952-8436-ebe3b1653436\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " Nov 01 20:19:58.101725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.101624 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-openshift-ca\") pod \"2eff0a38-18e7-4952-8436-ebe3b1653436\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " Nov 01 20:19:58.101725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.101663 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-xks-ca\") pod \"2eff0a38-18e7-4952-8436-ebe3b1653436\" (UID: \"2eff0a38-18e7-4952-8436-ebe3b1653436\") " Nov 01 20:19:58.102000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.101970 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-openshift-ca" pod "2eff0a38-18e7-4952-8436-ebe3b1653436" (UID: "2eff0a38-18e7-4952-8436-ebe3b1653436"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:58.103755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.103731 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eff0a38-18e7-4952-8436-ebe3b1653436-kube-api-access-9f9mf" pod "2eff0a38-18e7-4952-8436-ebe3b1653436" (UID: "2eff0a38-18e7-4952-8436-ebe3b1653436"). InnerVolumeSpecName "kube-api-access-9f9mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:58.103848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.103806 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-xks-ca" pod "2eff0a38-18e7-4952-8436-ebe3b1653436" (UID: "2eff0a38-18e7-4952-8436-ebe3b1653436"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:19:58.129371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.129129 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" podStartSLOduration=2.129109094 podStartE2EDuration="2.129109094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:57.967861745 +0000 UTC m=+1005.642121001" watchObservedRunningTime="2026-11-01 20:19:58.129109094 +0000 UTC m=+1005.803368349" Nov 01 20:19:58.203751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.203701 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f9mf\" (UniqueName: \"kubernetes.io/projected/2eff0a38-18e7-4952-8436-ebe3b1653436-kube-api-access-9f9mf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:58.203751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.203746 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:58.203751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.203756 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eff0a38-18e7-4952-8436-ebe3b1653436-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:58.237637 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.237536 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:19:58.850962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.850926 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" event={"ID":"2eff0a38-18e7-4952-8436-ebe3b1653436","Type":"ContainerDied","Data":"201c8c6023d4179b9875e4c387f9f311ee0bb77a6f7cd898af172c22b7a05df6"} Nov 01 20:19:58.851534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.850988 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="201c8c6023d4179b9875e4c387f9f311ee0bb77a6f7cd898af172c22b7a05df6" Nov 01 20:19:58.851534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.850937 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd" Nov 01 20:19:58.852822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.852795 5633 generic.go:356] "Generic (PLEG): container finished" podID="540c652b-3449-4fb9-ad3f-6191f42d95f3" containerID="07a98b8fec983208f50f2fb68e2f5f53452aa447a9ffe1cc910e97f0ecdc5b05" exitCode=0 Nov 01 20:19:58.852958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:19:58.852878 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" event={"ID":"540c652b-3449-4fb9-ad3f-6191f42d95f3","Type":"ContainerDied","Data":"07a98b8fec983208f50f2fb68e2f5f53452aa447a9ffe1cc910e97f0ecdc5b05"} Nov 01 20:20:00.029197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.029169 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:20:00.090294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.090262 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:00.129471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.129370 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540c652b-3449-4fb9-ad3f-6191f42d95f3-kube-api-access-fq98x\" (UniqueName: \"kubernetes.io/projected/540c652b-3449-4fb9-ad3f-6191f42d95f3-kube-api-access-fq98x\") pod \"540c652b-3449-4fb9-ad3f-6191f42d95f3\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " Nov 01 20:20:00.129471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.129418 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-openshift-ca\") pod \"540c652b-3449-4fb9-ad3f-6191f42d95f3\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " Nov 01 20:20:00.129705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.129518 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-xks-ca\") pod \"540c652b-3449-4fb9-ad3f-6191f42d95f3\" (UID: \"540c652b-3449-4fb9-ad3f-6191f42d95f3\") " Nov 01 20:20:00.129829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.129801 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-openshift-ca" pod "540c652b-3449-4fb9-ad3f-6191f42d95f3" (UID: "540c652b-3449-4fb9-ad3f-6191f42d95f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:00.130045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.130026 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:00.131748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.131725 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540c652b-3449-4fb9-ad3f-6191f42d95f3-kube-api-access-fq98x" pod "540c652b-3449-4fb9-ad3f-6191f42d95f3" (UID: "540c652b-3449-4fb9-ad3f-6191f42d95f3"). InnerVolumeSpecName "kube-api-access-fq98x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:00.131907 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.131888 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-xks-ca" pod "540c652b-3449-4fb9-ad3f-6191f42d95f3" (UID: "540c652b-3449-4fb9-ad3f-6191f42d95f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:00.231424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.231385 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq98x\" (UniqueName: \"kubernetes.io/projected/540c652b-3449-4fb9-ad3f-6191f42d95f3-kube-api-access-fq98x\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:00.231424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.231420 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c652b-3449-4fb9-ad3f-6191f42d95f3-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:00.870263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.870168 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" Nov 01 20:20:00.870263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.870211 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd" event={"ID":"540c652b-3449-4fb9-ad3f-6191f42d95f3","Type":"ContainerDied","Data":"eb9f93a95e7c21d7172c8c9f48945735a5778becbbf38b6284c44a6337f70f0a"} Nov 01 20:20:00.870517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.870272 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb9f93a95e7c21d7172c8c9f48945735a5778becbbf38b6284c44a6337f70f0a" Nov 01 20:20:00.968894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:00.968860 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:20:02.184427 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.184387 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz"] Nov 01 20:20:02.193430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.193394 5633 cpu_manager.go:414] "containerMap: removing container" podUID="2eff0a38-18e7-4952-8436-ebe3b1653436" containerName="revoke-keys" Nov 01 20:20:02.193673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.193653 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eff0a38-18e7-4952-8436-ebe3b1653436" containerName="revoke-keys" Nov 01 20:20:02.193814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.193802 5633 cpu_manager.go:414] "containerMap: removing container" podUID="540c652b-3449-4fb9-ad3f-6191f42d95f3" containerName="revoke-keys" Nov 01 20:20:02.193915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.193903 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540c652b-3449-4fb9-ad3f-6191f42d95f3" containerName="revoke-keys" Nov 01 20:20:02.194232 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.194216 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eff0a38-18e7-4952-8436-ebe3b1653436" containerName="revoke-keys" Nov 01 20:20:02.198413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.198380 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="540c652b-3449-4fb9-ad3f-6191f42d95f3" containerName="revoke-keys" Nov 01 20:20:02.224962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.224930 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:02.314360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.282808 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:20:02.314360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.283022 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:20:02.314360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.283650 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:20:02.314360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.283721 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:20:02.490435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.363662 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz"] Nov 01 20:20:02.604032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.587813 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:02.604032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.603675 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tn5\" (UniqueName: \"kubernetes.io/projected/dff23205-cd43-44f2-b390-d3a26d2543ce-kube-api-access-76tn5\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:02.604032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.603734 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:02.894551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.894447 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:02.894916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.894892 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tn5\" (UniqueName: \"kubernetes.io/projected/dff23205-cd43-44f2-b390-d3a26d2543ce-kube-api-access-76tn5\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:02.895092 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.895075 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:02.896682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:02.896653 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:03.033972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:03.033936 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:03.128360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:03.089261 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tn5\" (UniqueName: \"kubernetes.io/projected/dff23205-cd43-44f2-b390-d3a26d2543ce-kube-api-access-76tn5\") pod \"maas-api-revoke-sub-56417bbe-4s9wz\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:03.431361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:03.428600 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:03.530563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:03.530495 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:03.578549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:03.578475 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:03.962369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:03.959453 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:04.235490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:04.214847 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:04.548676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:04.547468 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 01 20:20:04.598361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:04.573357 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:20:04.704486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:04.601313 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:04.716438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:04.706915 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:20:04.990357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:04.971858 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:04.990357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:04.973124 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:20:05.141387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.139700 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz"] Nov 01 20:20:05.191379 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:05.168926 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff23205_cd43_44f2_b390_d3a26d2543ce.slice/crio-2806f50200826d0eb7bfc41989b1fe54e99d337fac387d9eb27693884067054f WatchSource:0}: Error finding container 2806f50200826d0eb7bfc41989b1fe54e99d337fac387d9eb27693884067054f: Status 404 returned error can't find the container with id 2806f50200826d0eb7bfc41989b1fe54e99d337fac387d9eb27693884067054f Nov 01 20:20:05.195212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.195182 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:20:05.239411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.239382 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:20:05.311492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.308559 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:20:05.396760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.395487 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" event={"ID":"dff23205-cd43-44f2-b390-d3a26d2543ce","Type":"ContainerStarted","Data":"2806f50200826d0eb7bfc41989b1fe54e99d337fac387d9eb27693884067054f"} Nov 01 20:20:05.563247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.563213 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:05.733213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.733166 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:05.899911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:05.899870 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:06.404453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:06.404420 5633 generic.go:356] "Generic (PLEG): container finished" podID="dff23205-cd43-44f2-b390-d3a26d2543ce" containerID="5e3a8777f12664f87440622c9d5d925d33d47e7a9936589c88adb622f9fb6755" exitCode=0 Nov 01 20:20:06.404656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:06.404476 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" event={"ID":"dff23205-cd43-44f2-b390-d3a26d2543ce","Type":"ContainerDied","Data":"5e3a8777f12664f87440622c9d5d925d33d47e7a9936589c88adb622f9fb6755"} Nov 01 20:20:07.584245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.584219 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:07.610067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.610032 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:07.692676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.692591 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-openshift-ca\") pod \"dff23205-cd43-44f2-b390-d3a26d2543ce\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " Nov 01 20:20:07.692841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.692689 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff23205-cd43-44f2-b390-d3a26d2543ce-kube-api-access-76tn5\" (UniqueName: \"kubernetes.io/projected/dff23205-cd43-44f2-b390-d3a26d2543ce-kube-api-access-76tn5\") pod \"dff23205-cd43-44f2-b390-d3a26d2543ce\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " Nov 01 20:20:07.692841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.692724 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-xks-ca\") pod \"dff23205-cd43-44f2-b390-d3a26d2543ce\" (UID: \"dff23205-cd43-44f2-b390-d3a26d2543ce\") " Nov 01 20:20:07.692958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.692936 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-openshift-ca" pod "dff23205-cd43-44f2-b390-d3a26d2543ce" (UID: "dff23205-cd43-44f2-b390-d3a26d2543ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:07.693262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.693239 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:07.694993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.694961 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-xks-ca" pod "dff23205-cd43-44f2-b390-d3a26d2543ce" (UID: "dff23205-cd43-44f2-b390-d3a26d2543ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:07.695091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.695030 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff23205-cd43-44f2-b390-d3a26d2543ce-kube-api-access-76tn5" pod "dff23205-cd43-44f2-b390-d3a26d2543ce" (UID: "dff23205-cd43-44f2-b390-d3a26d2543ce"). InnerVolumeSpecName "kube-api-access-76tn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:07.731490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.731455 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:07.794308 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.794270 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76tn5\" (UniqueName: \"kubernetes.io/projected/dff23205-cd43-44f2-b390-d3a26d2543ce-kube-api-access-76tn5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:07.794308 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:07.794303 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff23205-cd43-44f2-b390-d3a26d2543ce-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:08.443380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:08.434431 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" Nov 01 20:20:08.443380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:08.434616 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz" event={"ID":"dff23205-cd43-44f2-b390-d3a26d2543ce","Type":"ContainerDied","Data":"2806f50200826d0eb7bfc41989b1fe54e99d337fac387d9eb27693884067054f"} Nov 01 20:20:08.443380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:08.434676 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2806f50200826d0eb7bfc41989b1fe54e99d337fac387d9eb27693884067054f" Nov 01 20:20:10.517886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:10.517848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:11.738083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.738049 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r"] Nov 01 20:20:11.738687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.738671 5633 cpu_manager.go:414] "containerMap: removing container" podUID="dff23205-cd43-44f2-b390-d3a26d2543ce" containerName="revoke-keys" Nov 01 20:20:11.738748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.738689 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff23205-cd43-44f2-b390-d3a26d2543ce" containerName="revoke-keys" Nov 01 20:20:11.738801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.738792 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff23205-cd43-44f2-b390-d3a26d2543ce" containerName="revoke-keys" Nov 01 20:20:11.742639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.742622 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.799195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.799158 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r"] Nov 01 20:20:11.843177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.843134 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:20:11.843177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.843166 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:20:11.843444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.843235 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:20:11.844236 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.844217 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:20:11.856041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.856008 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fccvt\" (UniqueName: \"kubernetes.io/projected/b431aa23-1a34-4995-b07b-015e28965985-kube-api-access-fccvt\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.856207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.856125 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431aa23-1a34-4995-b07b-015e28965985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.856292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.856271 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431aa23-1a34-4995-b07b-015e28965985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.957302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.957256 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fccvt\" (UniqueName: \"kubernetes.io/projected/b431aa23-1a34-4995-b07b-015e28965985-kube-api-access-fccvt\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.957506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.957390 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431aa23-1a34-4995-b07b-015e28965985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.957564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.957515 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431aa23-1a34-4995-b07b-015e28965985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.958259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.958232 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431aa23-1a34-4995-b07b-015e28965985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:11.960476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:11.960454 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431aa23-1a34-4995-b07b-015e28965985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:12.014271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.014194 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fccvt\" (UniqueName: \"kubernetes.io/projected/b431aa23-1a34-4995-b07b-015e28965985-kube-api-access-fccvt\") pod \"maas-api-revoke-sub-87c649bc-wf27r\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:12.056518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.056488 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:12.429906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.429872 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r"] Nov 01 20:20:12.430410 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:12.430384 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb431aa23_1a34_4995_b07b_015e28965985.slice/crio-c378efb4e51c703112dae4f47c2d05c129841f0b074d9b48c361906ef1df8528 WatchSource:0}: Error finding container c378efb4e51c703112dae4f47c2d05c129841f0b074d9b48c361906ef1df8528: Status 404 returned error can't find the container with id c378efb4e51c703112dae4f47c2d05c129841f0b074d9b48c361906ef1df8528 Nov 01 20:20:12.470192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.470158 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" event={"ID":"b431aa23-1a34-4995-b07b-015e28965985","Type":"ContainerStarted","Data":"c378efb4e51c703112dae4f47c2d05c129841f0b074d9b48c361906ef1df8528"} Nov 01 20:20:12.587454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.587421 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67"] Nov 01 20:20:12.592945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.592924 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.648152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.648120 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67"] Nov 01 20:20:12.667110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.667062 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqb8\" (UniqueName: \"kubernetes.io/projected/9e93cf36-5d26-4e56-a2ed-a024973b99ed-kube-api-access-jpqb8\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.667290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.667137 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.667432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.667408 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.768780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.768741 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.769245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.768850 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqb8\" (UniqueName: \"kubernetes.io/projected/9e93cf36-5d26-4e56-a2ed-a024973b99ed-kube-api-access-jpqb8\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.769245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.768878 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.769510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.769491 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.771486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.771462 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.825901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.825866 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqb8\" (UniqueName: \"kubernetes.io/projected/9e93cf36-5d26-4e56-a2ed-a024973b99ed-kube-api-access-jpqb8\") pod \"maas-api-revoke-sub-ef75212c-fwl67\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.893920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.893880 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6"] Nov 01 20:20:12.898430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.898407 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:12.907363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.907304 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:12.955251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.955207 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6"] Nov 01 20:20:12.972375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.972264 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84986c63-697f-4e13-9268-2f9398b14229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:12.972521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.972441 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rrnv\" (UniqueName: \"kubernetes.io/projected/84986c63-697f-4e13-9268-2f9398b14229-kube-api-access-8rrnv\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:12.972521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:12.972480 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84986c63-697f-4e13-9268-2f9398b14229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.073810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.073725 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84986c63-697f-4e13-9268-2f9398b14229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.073955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.073813 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rrnv\" (UniqueName: \"kubernetes.io/projected/84986c63-697f-4e13-9268-2f9398b14229-kube-api-access-8rrnv\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.073955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.073839 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84986c63-697f-4e13-9268-2f9398b14229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.074440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.074406 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84986c63-697f-4e13-9268-2f9398b14229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.076429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.076397 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84986c63-697f-4e13-9268-2f9398b14229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.130203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.130168 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rrnv\" (UniqueName: \"kubernetes.io/projected/84986c63-697f-4e13-9268-2f9398b14229-kube-api-access-8rrnv\") pod \"maas-api-revoke-sub-1f67173f-rvzl6\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.212373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.212323 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:13.303134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.303100 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67"] Nov 01 20:20:13.366383 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:13.366261 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e93cf36_5d26_4e56_a2ed_a024973b99ed.slice/crio-09b0b151550f424d2b0b5642322d7a1d7461c264a9b6d0afa51b02ba89e5a71d WatchSource:0}: Error finding container 09b0b151550f424d2b0b5642322d7a1d7461c264a9b6d0afa51b02ba89e5a71d: Status 404 returned error can't find the container with id 09b0b151550f424d2b0b5642322d7a1d7461c264a9b6d0afa51b02ba89e5a71d Nov 01 20:20:13.475692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.475649 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" event={"ID":"9e93cf36-5d26-4e56-a2ed-a024973b99ed","Type":"ContainerStarted","Data":"09b0b151550f424d2b0b5642322d7a1d7461c264a9b6d0afa51b02ba89e5a71d"} Nov 01 20:20:13.477428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.477398 5633 generic.go:356] "Generic (PLEG): container finished" podID="b431aa23-1a34-4995-b07b-015e28965985" containerID="afd9cf34930e73c75da5b003c42cf4521707cc79d0cd0ebed6d734c07aa84f3d" exitCode=0 Nov 01 20:20:13.477548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.477489 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" event={"ID":"b431aa23-1a34-4995-b07b-015e28965985","Type":"ContainerDied","Data":"afd9cf34930e73c75da5b003c42cf4521707cc79d0cd0ebed6d734c07aa84f3d"} Nov 01 20:20:13.599841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:13.599812 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6"] Nov 01 20:20:13.604671 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:13.604637 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84986c63_697f_4e13_9268_2f9398b14229.slice/crio-5464dfa01a0bbacf00ee392aa79f6d5b423d266cf0404452d8f98b3431815d59 WatchSource:0}: Error finding container 5464dfa01a0bbacf00ee392aa79f6d5b423d266cf0404452d8f98b3431815d59: Status 404 returned error can't find the container with id 5464dfa01a0bbacf00ee392aa79f6d5b423d266cf0404452d8f98b3431815d59 Nov 01 20:20:14.485938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.485831 5633 generic.go:356] "Generic (PLEG): container finished" podID="9e93cf36-5d26-4e56-a2ed-a024973b99ed" containerID="927a942df956068bb8290c2eb2c001d5045bd14806b6016fb76bac10dd58f254" exitCode=0 Nov 01 20:20:14.486325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.485969 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" event={"ID":"9e93cf36-5d26-4e56-a2ed-a024973b99ed","Type":"ContainerDied","Data":"927a942df956068bb8290c2eb2c001d5045bd14806b6016fb76bac10dd58f254"} Nov 01 20:20:14.487887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.487855 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" event={"ID":"84986c63-697f-4e13-9268-2f9398b14229","Type":"ContainerStarted","Data":"8e239d8bde978e861716ad42f104c97ba2618bc41e5a2d3b40e198a2412a08d0"} Nov 01 20:20:14.488031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.487909 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" event={"ID":"84986c63-697f-4e13-9268-2f9398b14229","Type":"ContainerStarted","Data":"5464dfa01a0bbacf00ee392aa79f6d5b423d266cf0404452d8f98b3431815d59"} Nov 01 20:20:14.651105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.651078 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:14.710758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.710702 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" podStartSLOduration=2.710687846 podStartE2EDuration="2.710687846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:14.708595442 +0000 UTC m=+1022.382854697" watchObservedRunningTime="2026-11-01 20:20:14.710687846 +0000 UTC m=+1022.384947141" Nov 01 20:20:14.800617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.800584 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b431aa23-1a34-4995-b07b-015e28965985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431aa23-1a34-4995-b07b-015e28965985-maas-api-xks-ca\") pod \"b431aa23-1a34-4995-b07b-015e28965985\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " Nov 01 20:20:14.800821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.800633 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b431aa23-1a34-4995-b07b-015e28965985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431aa23-1a34-4995-b07b-015e28965985-maas-api-openshift-ca\") pod \"b431aa23-1a34-4995-b07b-015e28965985\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " Nov 01 20:20:14.800821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.800771 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b431aa23-1a34-4995-b07b-015e28965985-kube-api-access-fccvt\" (UniqueName: \"kubernetes.io/projected/b431aa23-1a34-4995-b07b-015e28965985-kube-api-access-fccvt\") pod \"b431aa23-1a34-4995-b07b-015e28965985\" (UID: \"b431aa23-1a34-4995-b07b-015e28965985\") " Nov 01 20:20:14.801081 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.801056 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b431aa23-1a34-4995-b07b-015e28965985-maas-api-openshift-ca" pod "b431aa23-1a34-4995-b07b-015e28965985" (UID: "b431aa23-1a34-4995-b07b-015e28965985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:14.803030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.803006 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b431aa23-1a34-4995-b07b-015e28965985-kube-api-access-fccvt" pod "b431aa23-1a34-4995-b07b-015e28965985" (UID: "b431aa23-1a34-4995-b07b-015e28965985"). InnerVolumeSpecName "kube-api-access-fccvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:14.803097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.803015 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b431aa23-1a34-4995-b07b-015e28965985-maas-api-xks-ca" pod "b431aa23-1a34-4995-b07b-015e28965985" (UID: "b431aa23-1a34-4995-b07b-015e28965985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:14.902663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.902565 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431aa23-1a34-4995-b07b-015e28965985-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:14.902663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.902610 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431aa23-1a34-4995-b07b-015e28965985-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:14.902663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:14.902627 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fccvt\" (UniqueName: \"kubernetes.io/projected/b431aa23-1a34-4995-b07b-015e28965985-kube-api-access-fccvt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:15.280201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.280161 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:15.497003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.496971 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" Nov 01 20:20:15.497476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.497011 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r" event={"ID":"b431aa23-1a34-4995-b07b-015e28965985","Type":"ContainerDied","Data":"c378efb4e51c703112dae4f47c2d05c129841f0b074d9b48c361906ef1df8528"} Nov 01 20:20:15.497476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.497057 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c378efb4e51c703112dae4f47c2d05c129841f0b074d9b48c361906ef1df8528" Nov 01 20:20:15.499870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.499838 5633 generic.go:356] "Generic (PLEG): container finished" podID="84986c63-697f-4e13-9268-2f9398b14229" containerID="8e239d8bde978e861716ad42f104c97ba2618bc41e5a2d3b40e198a2412a08d0" exitCode=0 Nov 01 20:20:15.499980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.499924 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" event={"ID":"84986c63-697f-4e13-9268-2f9398b14229","Type":"ContainerDied","Data":"8e239d8bde978e861716ad42f104c97ba2618bc41e5a2d3b40e198a2412a08d0"} Nov 01 20:20:15.657078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.657052 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:15.811941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.811843 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-xks-ca\") pod \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " Nov 01 20:20:15.812104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.811973 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-openshift-ca\") pod \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " Nov 01 20:20:15.812190 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.812103 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e93cf36-5d26-4e56-a2ed-a024973b99ed-kube-api-access-jpqb8\" (UniqueName: \"kubernetes.io/projected/9e93cf36-5d26-4e56-a2ed-a024973b99ed-kube-api-access-jpqb8\") pod \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\" (UID: \"9e93cf36-5d26-4e56-a2ed-a024973b99ed\") " Nov 01 20:20:15.812718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.812676 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-openshift-ca" pod "9e93cf36-5d26-4e56-a2ed-a024973b99ed" (UID: "9e93cf36-5d26-4e56-a2ed-a024973b99ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:15.816551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.816147 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e93cf36-5d26-4e56-a2ed-a024973b99ed-kube-api-access-jpqb8" pod "9e93cf36-5d26-4e56-a2ed-a024973b99ed" (UID: "9e93cf36-5d26-4e56-a2ed-a024973b99ed"). InnerVolumeSpecName "kube-api-access-jpqb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:15.816551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.816169 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-xks-ca" pod "9e93cf36-5d26-4e56-a2ed-a024973b99ed" (UID: "9e93cf36-5d26-4e56-a2ed-a024973b99ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:15.914445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.914408 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqb8\" (UniqueName: \"kubernetes.io/projected/9e93cf36-5d26-4e56-a2ed-a024973b99ed-kube-api-access-jpqb8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:15.914445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.914440 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:15.914445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:15.914449 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e93cf36-5d26-4e56-a2ed-a024973b99ed-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:16.163268 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:20:16.163180 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:20:16.508999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.508969 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" Nov 01 20:20:16.509394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.509015 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67" event={"ID":"9e93cf36-5d26-4e56-a2ed-a024973b99ed","Type":"ContainerDied","Data":"09b0b151550f424d2b0b5642322d7a1d7461c264a9b6d0afa51b02ba89e5a71d"} Nov 01 20:20:16.509394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.509062 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b0b151550f424d2b0b5642322d7a1d7461c264a9b6d0afa51b02ba89e5a71d" Nov 01 20:20:16.673772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.673743 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:16.825317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.825215 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84986c63-697f-4e13-9268-2f9398b14229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84986c63-697f-4e13-9268-2f9398b14229-maas-api-xks-ca\") pod \"84986c63-697f-4e13-9268-2f9398b14229\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " Nov 01 20:20:16.825490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.825437 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84986c63-697f-4e13-9268-2f9398b14229-kube-api-access-8rrnv\" (UniqueName: \"kubernetes.io/projected/84986c63-697f-4e13-9268-2f9398b14229-kube-api-access-8rrnv\") pod \"84986c63-697f-4e13-9268-2f9398b14229\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " Nov 01 20:20:16.825650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.825628 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84986c63-697f-4e13-9268-2f9398b14229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84986c63-697f-4e13-9268-2f9398b14229-maas-api-openshift-ca\") pod \"84986c63-697f-4e13-9268-2f9398b14229\" (UID: \"84986c63-697f-4e13-9268-2f9398b14229\") " Nov 01 20:20:16.826064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.826036 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84986c63-697f-4e13-9268-2f9398b14229-maas-api-openshift-ca" pod "84986c63-697f-4e13-9268-2f9398b14229" (UID: "84986c63-697f-4e13-9268-2f9398b14229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:16.827710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.827682 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84986c63-697f-4e13-9268-2f9398b14229-maas-api-xks-ca" pod "84986c63-697f-4e13-9268-2f9398b14229" (UID: "84986c63-697f-4e13-9268-2f9398b14229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:16.827810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.827778 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84986c63-697f-4e13-9268-2f9398b14229-kube-api-access-8rrnv" pod "84986c63-697f-4e13-9268-2f9398b14229" (UID: "84986c63-697f-4e13-9268-2f9398b14229"). InnerVolumeSpecName "kube-api-access-8rrnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:16.927777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.927719 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rrnv\" (UniqueName: \"kubernetes.io/projected/84986c63-697f-4e13-9268-2f9398b14229-kube-api-access-8rrnv\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:16.927777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.927768 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84986c63-697f-4e13-9268-2f9398b14229-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:16.927777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:16.927786 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84986c63-697f-4e13-9268-2f9398b14229-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:17.521222 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:17.521188 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" Nov 01 20:20:17.521222 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:17.521200 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6" event={"ID":"84986c63-697f-4e13-9268-2f9398b14229","Type":"ContainerDied","Data":"5464dfa01a0bbacf00ee392aa79f6d5b423d266cf0404452d8f98b3431815d59"} Nov 01 20:20:17.521679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:17.521259 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5464dfa01a0bbacf00ee392aa79f6d5b423d266cf0404452d8f98b3431815d59" Nov 01 20:20:20.124584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:20.124553 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:22.038374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:22.029487 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:22.205358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:22.204299 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:20:22.208450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:22.208406 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:22.310889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:22.310453 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:23.157355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.155957 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:23.205488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.205435 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:23.206513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.206487 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:20:23.229493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.229438 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:23.230129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.230105 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:20:23.318460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.318416 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:20:23.319061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.319035 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:20:23.712386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:23.633665 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:24.058416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:24.057446 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:20:24.252358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:24.222096 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:24.253447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:24.252943 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:24.352575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:24.333014 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:24.378354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:24.377990 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:20:24.408355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:24.384943 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:20:25.881886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:25.881842 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:27.612008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.611960 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7"] Nov 01 20:20:27.612652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612635 5633 cpu_manager.go:414] "containerMap: removing container" podUID="84986c63-697f-4e13-9268-2f9398b14229" containerName="revoke-keys" Nov 01 20:20:27.612652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612652 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84986c63-697f-4e13-9268-2f9398b14229" containerName="revoke-keys" Nov 01 20:20:27.612795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612682 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b431aa23-1a34-4995-b07b-015e28965985" containerName="revoke-keys" Nov 01 20:20:27.612795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612687 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b431aa23-1a34-4995-b07b-015e28965985" containerName="revoke-keys" Nov 01 20:20:27.612795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612716 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9e93cf36-5d26-4e56-a2ed-a024973b99ed" containerName="revoke-keys" Nov 01 20:20:27.612795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612723 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e93cf36-5d26-4e56-a2ed-a024973b99ed" containerName="revoke-keys" Nov 01 20:20:27.613015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612814 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="84986c63-697f-4e13-9268-2f9398b14229" containerName="revoke-keys" Nov 01 20:20:27.613015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612827 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b431aa23-1a34-4995-b07b-015e28965985" containerName="revoke-keys" Nov 01 20:20:27.613015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.612836 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e93cf36-5d26-4e56-a2ed-a024973b99ed" containerName="revoke-keys" Nov 01 20:20:27.616757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.616722 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.660980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.660940 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7"] Nov 01 20:20:27.710651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.706009 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:20:27.710651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.707468 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:20:27.710651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.707527 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:20:27.718725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.718694 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:20:27.746656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.746618 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.746835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.746669 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.746835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.746788 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5rx\" (UniqueName: \"kubernetes.io/projected/1739e3c3-b094-4fc9-bdfb-6b502b372e19-kube-api-access-zw5rx\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.847631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.847596 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.847823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.847649 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.847823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.847686 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5rx\" (UniqueName: \"kubernetes.io/projected/1739e3c3-b094-4fc9-bdfb-6b502b372e19-kube-api-access-zw5rx\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.848501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.848476 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.850227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.850197 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.904233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.904158 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5rx\" (UniqueName: \"kubernetes.io/projected/1739e3c3-b094-4fc9-bdfb-6b502b372e19-kube-api-access-zw5rx\") pod \"maas-api-revoke-sub-ed4ce23e-n7pm7\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:27.931225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:27.931185 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:28.321493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.321460 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7"] Nov 01 20:20:28.321948 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:28.321914 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1739e3c3_b094_4fc9_bdfb_6b502b372e19.slice/crio-2f468afc472ff61f4c12dbe686189d08ccf57edb50ee7ff3c1bf7dab5c87072b WatchSource:0}: Error finding container 2f468afc472ff61f4c12dbe686189d08ccf57edb50ee7ff3c1bf7dab5c87072b: Status 404 returned error can't find the container with id 2f468afc472ff61f4c12dbe686189d08ccf57edb50ee7ff3c1bf7dab5c87072b Nov 01 20:20:28.498989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.498955 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr"] Nov 01 20:20:28.504681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.504660 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.565613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.565584 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr"] Nov 01 20:20:28.657619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.657576 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhh5\" (UniqueName: \"kubernetes.io/projected/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-kube-api-access-7mhh5\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.658063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.657729 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.658063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.657793 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.759764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.759726 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhh5\" (UniqueName: \"kubernetes.io/projected/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-kube-api-access-7mhh5\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.759949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.759814 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.759949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.759842 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.760711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.760625 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.762491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.762470 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.816610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.816566 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhh5\" (UniqueName: \"kubernetes.io/projected/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-kube-api-access-7mhh5\") pod \"maas-api-revoke-sub-3e69b4eb-wz6fr\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:28.818083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:28.818062 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:29.203343 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:29.203301 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr"] Nov 01 20:20:29.210862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:29.210834 5633 generic.go:356] "Generic (PLEG): container finished" podID="1739e3c3-b094-4fc9-bdfb-6b502b372e19" containerID="404e790039987d781aa7e3ad31377b9588e236aad0d1df52aa853a2e7276f3ef" exitCode=0 Nov 01 20:20:29.211012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:29.210966 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" event={"ID":"1739e3c3-b094-4fc9-bdfb-6b502b372e19","Type":"ContainerDied","Data":"404e790039987d781aa7e3ad31377b9588e236aad0d1df52aa853a2e7276f3ef"} Nov 01 20:20:29.211056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:29.211028 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" event={"ID":"1739e3c3-b094-4fc9-bdfb-6b502b372e19","Type":"ContainerStarted","Data":"2f468afc472ff61f4c12dbe686189d08ccf57edb50ee7ff3c1bf7dab5c87072b"} Nov 01 20:20:29.229471 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:29.229437 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e18fa9_0d87_4620_9bc4_a97c8a3bd5c0.slice/crio-8997468d3d3a93ad075f95b6e9ace53b52b5be40f8dcb40e8b481107eac5deb5 WatchSource:0}: Error finding container 8997468d3d3a93ad075f95b6e9ace53b52b5be40f8dcb40e8b481107eac5deb5: Status 404 returned error can't find the container with id 8997468d3d3a93ad075f95b6e9ace53b52b5be40f8dcb40e8b481107eac5deb5 Nov 01 20:20:30.220030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.219984 5633 generic.go:356] "Generic (PLEG): container finished" podID="c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" containerID="309046e23c4cd528ffd784148e5e32cc284ae1503bc30ffd29504028fb3435e3" exitCode=0 Nov 01 20:20:30.220461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.220122 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" event={"ID":"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0","Type":"ContainerDied","Data":"309046e23c4cd528ffd784148e5e32cc284ae1503bc30ffd29504028fb3435e3"} Nov 01 20:20:30.220461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.220173 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" event={"ID":"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0","Type":"ContainerStarted","Data":"8997468d3d3a93ad075f95b6e9ace53b52b5be40f8dcb40e8b481107eac5deb5"} Nov 01 20:20:30.388791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.388752 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:30.483602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.483556 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-xks-ca\") pod \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " Nov 01 20:20:30.483792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.483685 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1739e3c3-b094-4fc9-bdfb-6b502b372e19-kube-api-access-zw5rx\" (UniqueName: \"kubernetes.io/projected/1739e3c3-b094-4fc9-bdfb-6b502b372e19-kube-api-access-zw5rx\") pod \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " Nov 01 20:20:30.483792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.483738 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-openshift-ca\") pod \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\" (UID: \"1739e3c3-b094-4fc9-bdfb-6b502b372e19\") " Nov 01 20:20:30.484182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.484152 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-openshift-ca" pod "1739e3c3-b094-4fc9-bdfb-6b502b372e19" (UID: "1739e3c3-b094-4fc9-bdfb-6b502b372e19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:30.485954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.485929 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-xks-ca" pod "1739e3c3-b094-4fc9-bdfb-6b502b372e19" (UID: "1739e3c3-b094-4fc9-bdfb-6b502b372e19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:30.486057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.486022 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1739e3c3-b094-4fc9-bdfb-6b502b372e19-kube-api-access-zw5rx" pod "1739e3c3-b094-4fc9-bdfb-6b502b372e19" (UID: "1739e3c3-b094-4fc9-bdfb-6b502b372e19"). InnerVolumeSpecName "kube-api-access-zw5rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:30.586488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.586455 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5rx\" (UniqueName: \"kubernetes.io/projected/1739e3c3-b094-4fc9-bdfb-6b502b372e19-kube-api-access-zw5rx\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:30.586488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.586489 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:30.586702 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.586502 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1739e3c3-b094-4fc9-bdfb-6b502b372e19-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:30.608882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:30.608844 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:31.051303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.051269 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp"] Nov 01 20:20:31.057491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.057459 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eb0aa14-sb6rp"] Nov 01 20:20:31.229140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.229115 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" Nov 01 20:20:31.229590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.229153 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7" event={"ID":"1739e3c3-b094-4fc9-bdfb-6b502b372e19","Type":"ContainerDied","Data":"2f468afc472ff61f4c12dbe686189d08ccf57edb50ee7ff3c1bf7dab5c87072b"} Nov 01 20:20:31.229590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.229247 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f468afc472ff61f4c12dbe686189d08ccf57edb50ee7ff3c1bf7dab5c87072b" Nov 01 20:20:31.419317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.419290 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:31.508237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.508201 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-xks-ca\") pod \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " Nov 01 20:20:31.508462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.508294 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-openshift-ca\") pod \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " Nov 01 20:20:31.508462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.508422 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-kube-api-access-7mhh5\" (UniqueName: \"kubernetes.io/projected/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-kube-api-access-7mhh5\") pod \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\" (UID: \"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0\") " Nov 01 20:20:31.509045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.509014 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-openshift-ca" pod "c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" (UID: "c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:31.511162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.511127 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-kube-api-access-7mhh5" pod "c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" (UID: "c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0"). InnerVolumeSpecName "kube-api-access-7mhh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:31.511270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.511211 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-xks-ca" pod "c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" (UID: "c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:31.609966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.609922 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:31.609966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.609958 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:31.609966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:31.609968 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mhh5\" (UniqueName: \"kubernetes.io/projected/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0-kube-api-access-7mhh5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:20:32.080348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.080287 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:32.241074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.241011 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" Nov 01 20:20:32.243173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.240997 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr" event={"ID":"c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0","Type":"ContainerDied","Data":"8997468d3d3a93ad075f95b6e9ace53b52b5be40f8dcb40e8b481107eac5deb5"} Nov 01 20:20:32.243173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.241216 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8997468d3d3a93ad075f95b6e9ace53b52b5be40f8dcb40e8b481107eac5deb5" Nov 01 20:20:32.250638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.250310 5633 generic.go:356] "Generic (PLEG): container finished" podID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerID="49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6" exitCode=0 Nov 01 20:20:32.250638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.250405 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" event={"ID":"61ede63f-d465-4d69-bda7-3800a6f7d614","Type":"ContainerDied","Data":"49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6"} Nov 01 20:20:32.250955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.250935 5633 scope.go:122] "RemoveContainer" containerID="49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6" Nov 01 20:20:32.252708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.252676 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:32.252787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.252757 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:32.252830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.252808 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:32.468924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:32.468877 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed715511-af16-4acb-9666-91a640c3e535" path="/var/lib/kubelet/pods/ed715511-af16-4acb-9666-91a640c3e535/volumes" Nov 01 20:20:33.276239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:33.276197 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" event={"ID":"61ede63f-d465-4d69-bda7-3800a6f7d614","Type":"ContainerStarted","Data":"11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a"} Nov 01 20:20:34.270426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:34.270382 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:20:34.271201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:34.271178 5633 patch_prober.go:29] interesting pod/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:20:34.271298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:34.271253 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 20:20:35.270913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:35.270876 5633 patch_prober.go:29] interesting pod/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:20:35.271423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:35.270948 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 20:20:35.343987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:35.343950 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:36.271400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:36.271313 5633 patch_prober.go:29] interesting pod/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:20:36.271818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:36.271427 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 20:20:37.270742 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:37.270704 5633 patch_prober.go:29] interesting pod/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:20:37.270930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:37.270764 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 20:20:37.645881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:37.645786 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:20:38.280357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:38.270828 5633 patch_prober.go:29] interesting pod/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 20:20:38.280357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:38.270915 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 20:20:39.049450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:39.048223 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:39.383619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:39.383486 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:40.013101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.013066 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:20:40.041359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.036760 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:20:40.054358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.052598 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:40.054358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.053372 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:20:40.058355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.056236 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 20:20:40.390535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.390430 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:40.433186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.416831 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:20:40.467373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:40.442019 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:41.136393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:41.115389 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:41.410546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:41.410438 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:20:41.488501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:41.488457 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:41.489163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:41.489141 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:20:41.848225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:41.848125 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:41.925371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:41.910985 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:20:41.925371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:41.923896 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:20:42.065359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:42.056482 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:42.489576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:42.489527 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:42.608662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:42.608632 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:20:44.971037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.970992 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652"] Nov 01 20:20:44.971884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.971863 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" containerName="revoke-keys" Nov 01 20:20:44.971935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.971888 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" containerName="revoke-keys" Nov 01 20:20:44.971969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.971946 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1739e3c3-b094-4fc9-bdfb-6b502b372e19" containerName="revoke-keys" Nov 01 20:20:44.971969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.971954 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1739e3c3-b094-4fc9-bdfb-6b502b372e19" containerName="revoke-keys" Nov 01 20:20:44.972130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.972118 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" containerName="revoke-keys" Nov 01 20:20:44.972167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.972140 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1739e3c3-b094-4fc9-bdfb-6b502b372e19" containerName="revoke-keys" Nov 01 20:20:44.977558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:44.977534 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.075621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.075585 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652"] Nov 01 20:20:45.115167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115131 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4deeb6d-3552-4d30-8e78-8e848757a623-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115199 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms79n\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-kube-api-access-ms79n\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115377 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115406 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115429 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115527 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115670 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115702 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.115759 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.115727 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.216737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.216694 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.216737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.216739 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.216760 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.216788 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.216902 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.216931 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.216963 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.217035 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4deeb6d-3552-4d30-8e78-8e848757a623-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.217116 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms79n\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-kube-api-access-ms79n\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.217182 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.217197 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.217257 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.217490 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.217800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.217783 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4deeb6d-3552-4d30-8e78-8e848757a623-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.219289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.219266 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.219556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.219538 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.273446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.273364 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms79n\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-kube-api-access-ms79n\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.273614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.273595 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.293305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.293267 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:45.724182 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:45.724148 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4deeb6d_3552_4d30_8e78_8e848757a623.slice/crio-252e5c54477ff308212208176f7914f759c9954bbd9b94552684442e53182521 WatchSource:0}: Error finding container 252e5c54477ff308212208176f7914f759c9954bbd9b94552684442e53182521: Status 404 returned error can't find the container with id 252e5c54477ff308212208176f7914f759c9954bbd9b94552684442e53182521 Nov 01 20:20:45.724546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.724524 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652"] Nov 01 20:20:45.726612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.726579 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:45.726710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.726645 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:45.726710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:45.726674 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:46.054307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:46.054210 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" event={"ID":"c4deeb6d-3552-4d30-8e78-8e848757a623","Type":"ContainerStarted","Data":"c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a"} Nov 01 20:20:46.054307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:46.054277 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" event={"ID":"c4deeb6d-3552-4d30-8e78-8e848757a623","Type":"ContainerStarted","Data":"252e5c54477ff308212208176f7914f759c9954bbd9b94552684442e53182521"} Nov 01 20:20:46.160627 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:20:46.160588 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:20:46.171070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:46.171009 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podStartSLOduration=2.170989738 podStartE2EDuration="2.170989738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:46.168799731 +0000 UTC m=+1053.843058986" watchObservedRunningTime="2026-11-01 20:20:46.170989738 +0000 UTC m=+1053.845248994" Nov 01 20:20:46.311618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:46.311500 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:46.313282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:46.313253 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 20:20:46.313405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:46.313321 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 20:20:46.603420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:46.602161 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:47.304360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:47.296822 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 20:20:47.304360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:47.296890 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 20:20:48.300017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:48.299979 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 20:20:48.300383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:48.300298 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 20:20:49.310506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:49.310463 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 20:20:49.310972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:49.310521 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 20:20:50.048231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:50.048197 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz"] Nov 01 20:20:50.113255 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:50.113219 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-899d0e66-bc5sz"] Nov 01 20:20:50.307897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:50.307800 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 20:20:50.308196 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:50.308162 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 20:20:50.477130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:50.477096 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91977758-d2a6-4f4b-a853-1111f5e4ade9" path="/var/lib/kubelet/pods/91977758-d2a6-4f4b-a853-1111f5e4ade9/volumes" Nov 01 20:20:50.776849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:50.776814 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:51.295938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:51.295902 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 20:20:51.296260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:51.296227 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 20:20:51.479822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:51.479787 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:52.298816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:52.298782 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:52.299314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:52.299292 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:52.300133 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:52.300115 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:20:55.033214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:55.033168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:55.357521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:55.357431 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:56.276521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.276480 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:56.448663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.448626 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:56.773247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.773211 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s"] Nov 01 20:20:56.777488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.777471 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.880296 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.880873 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6a675a-openshift-default-dockercfg-85chn\"" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.886790 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.886912 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e829d-0f1c-496e-bf40-f476c03bb904-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.887003 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.887036 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.887064 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.887106 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jm6\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-kube-api-access-48jm6\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.887165 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.887242 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.887286 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e829d-0f1c-496e-bf40-f476c03bb904-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.894767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.891988 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s"] Nov 01 20:20:56.988091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988050 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e829d-0f1c-496e-bf40-f476c03bb904-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988140 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988233 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e829d-0f1c-496e-bf40-f476c03bb904-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988302 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988374 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988406 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988452 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jm6\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-kube-api-access-48jm6\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988522 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988597 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988737 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988828 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.988957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988914 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.989014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.988964 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.989089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.989071 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e829d-0f1c-496e-bf40-f476c03bb904-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.990917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.990896 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e829d-0f1c-496e-bf40-f476c03bb904-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:56.990917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:56.990910 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:57.045851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.045769 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jm6\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-kube-api-access-48jm6\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:57.046906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.046872 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:57.091883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.091849 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:57.123520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.123484 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:20:57.524993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.524959 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s"] Nov 01 20:20:57.525419 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:20:57.525258 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599e829d_0f1c_496e_bf40_f476c03bb904.slice/crio-66920d4fa253307147942ca99b172658a86858b6c475593e351d210e67106754 WatchSource:0}: Error finding container 66920d4fa253307147942ca99b172658a86858b6c475593e351d210e67106754: Status 404 returned error can't find the container with id 66920d4fa253307147942ca99b172658a86858b6c475593e351d210e67106754 Nov 01 20:20:57.528227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.528190 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:57.528361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.528261 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:57.528361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:57.528291 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:20:58.215358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:58.214707 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" event={"ID":"599e829d-0f1c-496e-bf40-f476c03bb904","Type":"ContainerStarted","Data":"93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827"} Nov 01 20:20:58.215358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:58.214783 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" event={"ID":"599e829d-0f1c-496e-bf40-f476c03bb904","Type":"ContainerStarted","Data":"66920d4fa253307147942ca99b172658a86858b6c475593e351d210e67106754"} Nov 01 20:20:59.107359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:59.097998 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:59.138358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:59.137699 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:59.262122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:59.262091 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:59.283680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:59.283647 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:20:59.380425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:20:59.376454 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" podStartSLOduration=3.376429871 podStartE2EDuration="3.376429871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:58.389383716 +0000 UTC m=+1066.063642971" watchObservedRunningTime="2026-11-01 20:20:59.376429871 +0000 UTC m=+1067.050689128" Nov 01 20:21:00.638360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:00.612739 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:01.523080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:01.523045 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:01.687700 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:01.687661 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:01.819410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:01.819282 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:04.736201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:04.736167 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:04.941954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:04.941914 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:05.157008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.156925 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph"] Nov 01 20:21:05.162618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.162584 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.206270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.206232 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20986a23-c21b-4951-98be-b22b660f14b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.206434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.206367 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tq6w\" (UniqueName: \"kubernetes.io/projected/20986a23-c21b-4951-98be-b22b660f14b2-kube-api-access-4tq6w\") pod \"payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.220150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.220118 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph"] Nov 01 20:21:05.263098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.263062 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6a675a\"" Nov 01 20:21:05.316747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.316705 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20986a23-c21b-4951-98be-b22b660f14b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.332571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.332530 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tq6w\" (UniqueName: \"kubernetes.io/projected/20986a23-c21b-4951-98be-b22b660f14b2-kube-api-access-4tq6w\") pod \"payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.345606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.318064 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20986a23-c21b-4951-98be-b22b660f14b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.398301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.398274 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tq6w\" (UniqueName: \"kubernetes.io/projected/20986a23-c21b-4951-98be-b22b660f14b2-kube-api-access-4tq6w\") pod \"payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.431073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.430928 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:05.478775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.478743 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:05.588270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.588187 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:05.863912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:05.863878 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph"] Nov 01 20:21:05.864385 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:21:05.863987 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20986a23_c21b_4951_98be_b22b660f14b2.slice/crio-95f4ef267954107da19d9c94b5d9043fe3516eb48a4cc08f6c1857c9411163f3 WatchSource:0}: Error finding container 95f4ef267954107da19d9c94b5d9043fe3516eb48a4cc08f6c1857c9411163f3: Status 404 returned error can't find the container with id 95f4ef267954107da19d9c94b5d9043fe3516eb48a4cc08f6c1857c9411163f3 Nov 01 20:21:06.405395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:06.405357 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" event={"ID":"20986a23-c21b-4951-98be-b22b660f14b2","Type":"ContainerStarted","Data":"21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022"} Nov 01 20:21:06.405584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:06.405419 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" event={"ID":"20986a23-c21b-4951-98be-b22b660f14b2","Type":"ContainerStarted","Data":"95f4ef267954107da19d9c94b5d9043fe3516eb48a4cc08f6c1857c9411163f3"} Nov 01 20:21:06.405584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:06.405447 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:06.519013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:06.518958 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" podStartSLOduration=1.518941264 podStartE2EDuration="1.518941264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:06.517300712 +0000 UTC m=+1074.191559976" watchObservedRunningTime="2026-11-01 20:21:06.518941264 +0000 UTC m=+1074.193200519" Nov 01 20:21:06.771915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:06.771875 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:08.049097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:08.049060 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5"] Nov 01 20:21:08.097940 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:08.097900 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ab7d8-27rp5"] Nov 01 20:21:08.468187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:08.468153 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dcb2d15-cee3-4418-ad33-7ca7d7f0c256" path="/var/lib/kubelet/pods/5dcb2d15-cee3-4418-ad33-7ca7d7f0c256/volumes" Nov 01 20:21:09.879861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:09.879823 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:11.516410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:11.516375 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:11.961471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:11.961444 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:21:15.925043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:15.925004 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5"] Nov 01 20:21:15.933974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:15.933939 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:15.974283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:15.974246 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:15.974470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:15.974296 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:15.986407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:15.986374 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5"] Nov 01 20:21:16.030731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.030702 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:21:16.030937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.030743 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:21:16.030937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.030921 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:21:16.037199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.037177 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:21:16.075941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.075905 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66bn\" (UniqueName: \"kubernetes.io/projected/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-kube-api-access-x66bn\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.076122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.076017 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.076122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.076046 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.076904 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.076827 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.078549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.078528 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.158968 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:21:16.158932 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:21:16.177684 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.177597 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66bn\" (UniqueName: \"kubernetes.io/projected/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-kube-api-access-x66bn\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.202848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.202813 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:16.236583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.236553 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66bn\" (UniqueName: \"kubernetes.io/projected/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-kube-api-access-x66bn\") pod \"maas-api-revoke-sub-0342a788-zm7z5\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.254165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.254135 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:16.647555 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:21:16.647517 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9434a3e_0ec3_4dd5_bb4c_d3c526b29dbf.slice/crio-b15444a3f2eb4b3b9485478cc7861db833d29f2b912e17a883fa65d9467fecea WatchSource:0}: Error finding container b15444a3f2eb4b3b9485478cc7861db833d29f2b912e17a883fa65d9467fecea: Status 404 returned error can't find the container with id b15444a3f2eb4b3b9485478cc7861db833d29f2b912e17a883fa65d9467fecea Nov 01 20:21:16.647936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:16.647915 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5"] Nov 01 20:21:17.416911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:17.416882 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 20:21:17.493141 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:17.493103 5633 generic.go:356] "Generic (PLEG): container finished" podID="b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" containerID="a24b0eb11ff231831a75a1457fbd39a5022cdd000afc072e06923d049ca465cb" exitCode=0 Nov 01 20:21:17.493319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:17.493180 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" event={"ID":"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf","Type":"ContainerDied","Data":"a24b0eb11ff231831a75a1457fbd39a5022cdd000afc072e06923d049ca465cb"} Nov 01 20:21:17.493319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:17.493239 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" event={"ID":"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf","Type":"ContainerStarted","Data":"b15444a3f2eb4b3b9485478cc7861db833d29f2b912e17a883fa65d9467fecea"} Nov 01 20:21:18.674568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.674539 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:18.713263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.713230 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-kube-api-access-x66bn\" (UniqueName: \"kubernetes.io/projected/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-kube-api-access-x66bn\") pod \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " Nov 01 20:21:18.713466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.713423 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-openshift-ca\") pod \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " Nov 01 20:21:18.713466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.713450 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-xks-ca\") pod \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\" (UID: \"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf\") " Nov 01 20:21:18.713831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.713790 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-openshift-ca" pod "b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" (UID: "b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:18.714067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.714048 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:18.715755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.715730 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-kube-api-access-x66bn" pod "b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" (UID: "b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf"). InnerVolumeSpecName "kube-api-access-x66bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:18.715857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.715775 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-xks-ca" pod "b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" (UID: "b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:21:18.815169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.815073 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66bn\" (UniqueName: \"kubernetes.io/projected/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-kube-api-access-x66bn\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:18.815169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:18.815105 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:19.534359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:19.531732 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" event={"ID":"b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf","Type":"ContainerDied","Data":"b15444a3f2eb4b3b9485478cc7861db833d29f2b912e17a883fa65d9467fecea"} Nov 01 20:21:19.534359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:19.531804 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b15444a3f2eb4b3b9485478cc7861db833d29f2b912e17a883fa65d9467fecea" Nov 01 20:21:19.534359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:19.531873 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5" Nov 01 20:21:21.495559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:21.495434 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:21.714092 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:21.714053 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:21.926544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:21.926438 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:21.951476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:21.951433 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:22.034967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.034920 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:22.441584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.441538 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:21:22.441891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.441861 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:22.498970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.498926 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:21:22.531085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.531050 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:21:22.953357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.951170 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:21:22.953357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.951823 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:22.953357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.952159 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:21:22.953357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:22.952218 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:21:23.523167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:23.523107 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:21:23.558823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:23.557250 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:23.684280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:23.684247 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:21:23.754045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:23.753879 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:21:23.766053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:23.766017 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:21:26.914216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:26.914180 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:31.277691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.277657 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:21:31.320632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.320591 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l"] Nov 01 20:21:31.321248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.321234 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" containerName="revoke-keys" Nov 01 20:21:31.321299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.321249 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" containerName="revoke-keys" Nov 01 20:21:31.321380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.321373 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" containerName="revoke-keys" Nov 01 20:21:31.325284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.325264 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.372604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.372566 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.372790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.372628 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.373050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.372996 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhnv\" (UniqueName: \"kubernetes.io/projected/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-kube-api-access-wzhnv\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.377443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.377408 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:21:31.377566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.377448 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:21:31.377566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.377490 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:21:31.377566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.377532 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:21:31.431421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.431385 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l"] Nov 01 20:21:31.473872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.473834 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.474043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.473897 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.474090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.474058 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhnv\" (UniqueName: \"kubernetes.io/projected/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-kube-api-access-wzhnv\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.474659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.474633 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.476380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.476352 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.530696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.530625 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhnv\" (UniqueName: \"kubernetes.io/projected/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-kube-api-access-wzhnv\") pod \"maas-api-revoke-sub-1ce44211-jd66l\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.639604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.639562 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f"] Nov 01 20:21:31.639804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.639571 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:31.645167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.645142 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.651083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.651051 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:31.676753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.676719 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.676913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.676819 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8kz\" (UniqueName: \"kubernetes.io/projected/3c8861be-1d5f-4567-9618-015efeddaef3-kube-api-access-7m8kz\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.676913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.676875 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.698218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.698184 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f"] Nov 01 20:21:31.797409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.789392 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.797409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.789458 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8kz\" (UniqueName: \"kubernetes.io/projected/3c8861be-1d5f-4567-9618-015efeddaef3-kube-api-access-7m8kz\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.797409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.789529 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.825360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.807938 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.825360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.811751 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.848666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.848612 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:31.877357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.872042 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8kz\" (UniqueName: \"kubernetes.io/projected/3c8861be-1d5f-4567-9618-015efeddaef3-kube-api-access-7m8kz\") pod \"maas-api-revoke-sub-243acbce-7vz2f\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:31.986341 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:31.986291 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:32.091230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:32.091192 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l"] Nov 01 20:21:32.431466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:32.431434 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f"] Nov 01 20:21:32.431858 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:21:32.431834 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8861be_1d5f_4567_9618_015efeddaef3.slice/crio-e8296a791184f560e06808c1604bd91d346f4f5b3eba72f7bacfec5786f8db57 WatchSource:0}: Error finding container e8296a791184f560e06808c1604bd91d346f4f5b3eba72f7bacfec5786f8db57: Status 404 returned error can't find the container with id e8296a791184f560e06808c1604bd91d346f4f5b3eba72f7bacfec5786f8db57 Nov 01 20:21:32.876638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:32.876602 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" event={"ID":"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b","Type":"ContainerStarted","Data":"62bba7dd27829c3ebe18048ae6f13b3c4db153efac3f26495765eda6b4c0370a"} Nov 01 20:21:32.876839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:32.876662 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" event={"ID":"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b","Type":"ContainerStarted","Data":"1f6f616cbad8fde475c777d97e66fb747b627df06d27c4ef989b41584936a307"} Nov 01 20:21:32.878487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:32.878459 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" event={"ID":"3c8861be-1d5f-4567-9618-015efeddaef3","Type":"ContainerStarted","Data":"e0007837601a6589ac1176ea5810ccacec97493e22f4b58cc86ebf451171e7e0"} Nov 01 20:21:32.878627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:32.878502 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" event={"ID":"3c8861be-1d5f-4567-9618-015efeddaef3","Type":"ContainerStarted","Data":"e8296a791184f560e06808c1604bd91d346f4f5b3eba72f7bacfec5786f8db57"} Nov 01 20:21:32.997691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:32.997626 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" podStartSLOduration=1.997606898 podStartE2EDuration="1.997606898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:32.99438944 +0000 UTC m=+1100.668648692" watchObservedRunningTime="2026-11-01 20:21:32.997606898 +0000 UTC m=+1100.671866154" Nov 01 20:21:33.109639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:33.109577 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" podStartSLOduration=2.109562695 podStartE2EDuration="2.109562695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:33.105819329 +0000 UTC m=+1100.780078584" watchObservedRunningTime="2026-11-01 20:21:33.109562695 +0000 UTC m=+1100.783821950" Nov 01 20:21:33.888505 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:33.888464 5633 generic.go:356] "Generic (PLEG): container finished" podID="cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" containerID="62bba7dd27829c3ebe18048ae6f13b3c4db153efac3f26495765eda6b4c0370a" exitCode=0 Nov 01 20:21:33.888887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:33.888549 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" event={"ID":"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b","Type":"ContainerDied","Data":"62bba7dd27829c3ebe18048ae6f13b3c4db153efac3f26495765eda6b4c0370a"} Nov 01 20:21:33.890404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:33.890377 5633 generic.go:356] "Generic (PLEG): container finished" podID="3c8861be-1d5f-4567-9618-015efeddaef3" containerID="e0007837601a6589ac1176ea5810ccacec97493e22f4b58cc86ebf451171e7e0" exitCode=0 Nov 01 20:21:33.890543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:33.890473 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" event={"ID":"3c8861be-1d5f-4567-9618-015efeddaef3","Type":"ContainerDied","Data":"e0007837601a6589ac1176ea5810ccacec97493e22f4b58cc86ebf451171e7e0"} Nov 01 20:21:35.070923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.070891 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:35.096071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.096047 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:35.146518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146475 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-openshift-ca\") pod \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " Nov 01 20:21:35.146518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146518 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-openshift-ca\") pod \"3c8861be-1d5f-4567-9618-015efeddaef3\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " Nov 01 20:21:35.146761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146554 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-xks-ca\") pod \"3c8861be-1d5f-4567-9618-015efeddaef3\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " Nov 01 20:21:35.146761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146681 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8861be-1d5f-4567-9618-015efeddaef3-kube-api-access-7m8kz\" (UniqueName: \"kubernetes.io/projected/3c8861be-1d5f-4567-9618-015efeddaef3-kube-api-access-7m8kz\") pod \"3c8861be-1d5f-4567-9618-015efeddaef3\" (UID: \"3c8861be-1d5f-4567-9618-015efeddaef3\") " Nov 01 20:21:35.146864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146799 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-xks-ca\") pod \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " Nov 01 20:21:35.146921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146868 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-openshift-ca" pod "cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" (UID: "cc9d59d2-d4be-4142-bf51-1e7c4ed9217b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:35.146921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146876 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-openshift-ca" pod "3c8861be-1d5f-4567-9618-015efeddaef3" (UID: "3c8861be-1d5f-4567-9618-015efeddaef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:35.146921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.146876 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-kube-api-access-wzhnv\" (UniqueName: \"kubernetes.io/projected/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-kube-api-access-wzhnv\") pod \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\" (UID: \"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b\") " Nov 01 20:21:35.147755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.147726 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:35.147894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.147758 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:35.149036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.148972 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-xks-ca" pod "3c8861be-1d5f-4567-9618-015efeddaef3" (UID: "3c8861be-1d5f-4567-9618-015efeddaef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:21:35.149036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.148987 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8861be-1d5f-4567-9618-015efeddaef3-kube-api-access-7m8kz" pod "3c8861be-1d5f-4567-9618-015efeddaef3" (UID: "3c8861be-1d5f-4567-9618-015efeddaef3"). InnerVolumeSpecName "kube-api-access-7m8kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:35.149036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.149003 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-xks-ca" pod "cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" (UID: "cc9d59d2-d4be-4142-bf51-1e7c4ed9217b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:21:35.149192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.149045 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-kube-api-access-wzhnv" pod "cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" (UID: "cc9d59d2-d4be-4142-bf51-1e7c4ed9217b"). InnerVolumeSpecName "kube-api-access-wzhnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:35.249421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.249390 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:35.249421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.249418 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzhnv\" (UniqueName: \"kubernetes.io/projected/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b-kube-api-access-wzhnv\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:35.249421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.249430 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8861be-1d5f-4567-9618-015efeddaef3-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:35.249665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.249438 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m8kz\" (UniqueName: \"kubernetes.io/projected/3c8861be-1d5f-4567-9618-015efeddaef3-kube-api-access-7m8kz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:35.907864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.907833 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" Nov 01 20:21:35.907864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.907854 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f" event={"ID":"3c8861be-1d5f-4567-9618-015efeddaef3","Type":"ContainerDied","Data":"e8296a791184f560e06808c1604bd91d346f4f5b3eba72f7bacfec5786f8db57"} Nov 01 20:21:35.908140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.907903 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8296a791184f560e06808c1604bd91d346f4f5b3eba72f7bacfec5786f8db57" Nov 01 20:21:35.909568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.909547 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" Nov 01 20:21:35.909680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.909586 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l" event={"ID":"cc9d59d2-d4be-4142-bf51-1e7c4ed9217b","Type":"ContainerDied","Data":"1f6f616cbad8fde475c777d97e66fb747b627df06d27c4ef989b41584936a307"} Nov 01 20:21:35.909680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:35.909628 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f6f616cbad8fde475c777d97e66fb747b627df06d27c4ef989b41584936a307" Nov 01 20:21:36.393380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:36.393344 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:40.932759 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:40.932724 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:41.052895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:41.052860 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb"] Nov 01 20:21:41.090388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:41.090307 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:41.093426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:41.093407 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a1b989-4wlbb"] Nov 01 20:21:41.209749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:41.209668 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:42.467625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:42.467594 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e941ee79-7eb1-4511-b7f2-705c3f04c7a7" path="/var/lib/kubelet/pods/e941ee79-7eb1-4511-b7f2-705c3f04c7a7/volumes" Nov 01 20:21:46.243268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:46.243234 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:46.323670 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:21:46.323626 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:21:47.472493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.472457 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:47.629078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.629044 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z"] Nov 01 20:21:47.629696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.629680 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3c8861be-1d5f-4567-9618-015efeddaef3" containerName="revoke-keys" Nov 01 20:21:47.629745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.629697 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8861be-1d5f-4567-9618-015efeddaef3" containerName="revoke-keys" Nov 01 20:21:47.629745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.629710 5633 cpu_manager.go:414] "containerMap: removing container" podUID="cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" containerName="revoke-keys" Nov 01 20:21:47.629745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.629715 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" containerName="revoke-keys" Nov 01 20:21:47.629875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.629864 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8861be-1d5f-4567-9618-015efeddaef3" containerName="revoke-keys" Nov 01 20:21:47.629928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.629876 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" containerName="revoke-keys" Nov 01 20:21:47.634696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.634671 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.689728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.689697 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z"] Nov 01 20:21:47.734268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.734178 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:21:47.734268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.734239 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:21:47.734268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.734245 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:21:47.734741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.734716 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.734852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.734834 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.735035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.735018 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8x5t\" (UniqueName: \"kubernetes.io/projected/394f76af-9fca-496a-90c1-c49051bac02a-kube-api-access-w8x5t\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.735414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.735396 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:21:47.836283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.836251 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.836517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.836342 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.836567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.836523 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8x5t\" (UniqueName: \"kubernetes.io/projected/394f76af-9fca-496a-90c1-c49051bac02a-kube-api-access-w8x5t\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.837198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.837171 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.839235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.839205 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.893273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.893236 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8x5t\" (UniqueName: \"kubernetes.io/projected/394f76af-9fca-496a-90c1-c49051bac02a-kube-api-access-w8x5t\") pod \"maas-api-revoke-sub-4b0cc270-d5c9z\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:47.948573 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:47.948533 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:48.334822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:48.334778 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z"] Nov 01 20:21:48.334983 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:21:48.334848 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394f76af_9fca_496a_90c1_c49051bac02a.slice/crio-d894e3f9b8b833e0aac93e2c03283c6f8c15f7d61003c1c972f0cc8abb280bb5 WatchSource:0}: Error finding container d894e3f9b8b833e0aac93e2c03283c6f8c15f7d61003c1c972f0cc8abb280bb5: Status 404 returned error can't find the container with id d894e3f9b8b833e0aac93e2c03283c6f8c15f7d61003c1c972f0cc8abb280bb5 Nov 01 20:21:49.048127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:49.048087 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" event={"ID":"394f76af-9fca-496a-90c1-c49051bac02a","Type":"ContainerStarted","Data":"86dbceaf602e188d1f6a8a1a3e7b543aebfb8cf757638a849897a93d74245ef1"} Nov 01 20:21:49.048560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:49.048148 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" event={"ID":"394f76af-9fca-496a-90c1-c49051bac02a","Type":"ContainerStarted","Data":"d894e3f9b8b833e0aac93e2c03283c6f8c15f7d61003c1c972f0cc8abb280bb5"} Nov 01 20:21:49.165688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:49.165632 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" podStartSLOduration=2.165615966 podStartE2EDuration="2.165615966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:49.162655346 +0000 UTC m=+1116.836914598" watchObservedRunningTime="2026-11-01 20:21:49.165615966 +0000 UTC m=+1116.839875223" Nov 01 20:21:50.057887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:50.057849 5633 generic.go:356] "Generic (PLEG): container finished" podID="394f76af-9fca-496a-90c1-c49051bac02a" containerID="86dbceaf602e188d1f6a8a1a3e7b543aebfb8cf757638a849897a93d74245ef1" exitCode=0 Nov 01 20:21:50.058292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:50.057939 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" event={"ID":"394f76af-9fca-496a-90c1-c49051bac02a","Type":"ContainerDied","Data":"86dbceaf602e188d1f6a8a1a3e7b543aebfb8cf757638a849897a93d74245ef1"} Nov 01 20:21:50.978864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:50.978827 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:51.226889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.226863 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:51.384984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.384874 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394f76af-9fca-496a-90c1-c49051bac02a-kube-api-access-w8x5t\" (UniqueName: \"kubernetes.io/projected/394f76af-9fca-496a-90c1-c49051bac02a-kube-api-access-w8x5t\") pod \"394f76af-9fca-496a-90c1-c49051bac02a\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " Nov 01 20:21:51.384984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.384949 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-xks-ca\") pod \"394f76af-9fca-496a-90c1-c49051bac02a\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " Nov 01 20:21:51.384984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.384980 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-openshift-ca\") pod \"394f76af-9fca-496a-90c1-c49051bac02a\" (UID: \"394f76af-9fca-496a-90c1-c49051bac02a\") " Nov 01 20:21:51.385782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.385752 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-openshift-ca" pod "394f76af-9fca-496a-90c1-c49051bac02a" (UID: "394f76af-9fca-496a-90c1-c49051bac02a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:51.387601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.387572 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-xks-ca" pod "394f76af-9fca-496a-90c1-c49051bac02a" (UID: "394f76af-9fca-496a-90c1-c49051bac02a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:21:51.393588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.393549 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394f76af-9fca-496a-90c1-c49051bac02a-kube-api-access-w8x5t" pod "394f76af-9fca-496a-90c1-c49051bac02a" (UID: "394f76af-9fca-496a-90c1-c49051bac02a"). InnerVolumeSpecName "kube-api-access-w8x5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:51.396793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.396760 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf"] Nov 01 20:21:51.403710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.403681 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:51.487291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.487244 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8x5t\" (UniqueName: \"kubernetes.io/projected/394f76af-9fca-496a-90c1-c49051bac02a-kube-api-access-w8x5t\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:51.487291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.487286 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:51.487291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:51.487300 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394f76af-9fca-496a-90c1-c49051bac02a-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:52.079573 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.079541 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" Nov 01 20:21:52.079764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.079580 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z" event={"ID":"394f76af-9fca-496a-90c1-c49051bac02a","Type":"ContainerDied","Data":"d894e3f9b8b833e0aac93e2c03283c6f8c15f7d61003c1c972f0cc8abb280bb5"} Nov 01 20:21:52.079764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.079633 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d894e3f9b8b833e0aac93e2c03283c6f8c15f7d61003c1c972f0cc8abb280bb5" Nov 01 20:21:52.360394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.360287 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd"] Nov 01 20:21:52.360948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.360933 5633 cpu_manager.go:414] "containerMap: removing container" podUID="394f76af-9fca-496a-90c1-c49051bac02a" containerName="revoke-keys" Nov 01 20:21:52.360992 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.360951 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394f76af-9fca-496a-90c1-c49051bac02a" containerName="revoke-keys" Nov 01 20:21:52.361064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.361056 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="394f76af-9fca-496a-90c1-c49051bac02a" containerName="revoke-keys" Nov 01 20:21:52.365037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.365019 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.420499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.420460 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd"] Nov 01 20:21:52.421613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.421585 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:21:52.421748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.421617 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:21:52.421748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.421594 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:21:52.421748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.421624 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:21:52.499652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.499612 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.499841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.499689 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpz2g\" (UniqueName: \"kubernetes.io/projected/67a76671-86e0-4a7e-afa6-6d0649efceb2-kube-api-access-dpz2g\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.499841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.499750 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.601469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.601440 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpz2g\" (UniqueName: \"kubernetes.io/projected/67a76671-86e0-4a7e-afa6-6d0649efceb2-kube-api-access-dpz2g\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.601469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.601474 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.601713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.601636 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.602210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.602187 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.604065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.604038 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.658244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.658164 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpz2g\" (UniqueName: \"kubernetes.io/projected/67a76671-86e0-4a7e-afa6-6d0649efceb2-kube-api-access-dpz2g\") pod \"maas-api-revoke-sub-c5732ec0-5zlkd\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:52.679762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:52.679485 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:53.046988 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:53.046943 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl"] Nov 01 20:21:53.051174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:53.051144 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ab432-ch5vl"] Nov 01 20:21:53.060783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:53.060754 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd"] Nov 01 20:21:53.102045 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:21:53.102010 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a76671_86e0_4a7e_afa6_6d0649efceb2.slice/crio-ee38bc776ba90588799dd215be80b610533ac7e91c2cc2a14be770944748f2b8 WatchSource:0}: Error finding container ee38bc776ba90588799dd215be80b610533ac7e91c2cc2a14be770944748f2b8: Status 404 returned error can't find the container with id ee38bc776ba90588799dd215be80b610533ac7e91c2cc2a14be770944748f2b8 Nov 01 20:21:54.098031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:54.097997 5633 generic.go:356] "Generic (PLEG): container finished" podID="67a76671-86e0-4a7e-afa6-6d0649efceb2" containerID="79dd18c8ec01da41073ecf1db0a2099d8f9ff3579e211bcdb4d318946da1c833" exitCode=0 Nov 01 20:21:54.098502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:54.098083 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" event={"ID":"67a76671-86e0-4a7e-afa6-6d0649efceb2","Type":"ContainerDied","Data":"79dd18c8ec01da41073ecf1db0a2099d8f9ff3579e211bcdb4d318946da1c833"} Nov 01 20:21:54.098502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:54.098137 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" event={"ID":"67a76671-86e0-4a7e-afa6-6d0649efceb2","Type":"ContainerStarted","Data":"ee38bc776ba90588799dd215be80b610533ac7e91c2cc2a14be770944748f2b8"} Nov 01 20:21:54.468624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:54.468591 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f50230-bb2d-4a1f-a2ef-7f1361480592" path="/var/lib/kubelet/pods/09f50230-bb2d-4a1f-a2ef-7f1361480592/volumes" Nov 01 20:21:55.265563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.265532 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:55.336089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.336056 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-openshift-ca\") pod \"67a76671-86e0-4a7e-afa6-6d0649efceb2\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " Nov 01 20:21:55.336275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.336232 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-xks-ca\") pod \"67a76671-86e0-4a7e-afa6-6d0649efceb2\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " Nov 01 20:21:55.336634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.336281 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a76671-86e0-4a7e-afa6-6d0649efceb2-kube-api-access-dpz2g\" (UniqueName: \"kubernetes.io/projected/67a76671-86e0-4a7e-afa6-6d0649efceb2-kube-api-access-dpz2g\") pod \"67a76671-86e0-4a7e-afa6-6d0649efceb2\" (UID: \"67a76671-86e0-4a7e-afa6-6d0649efceb2\") " Nov 01 20:21:55.336634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.336471 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-openshift-ca" pod "67a76671-86e0-4a7e-afa6-6d0649efceb2" (UID: "67a76671-86e0-4a7e-afa6-6d0649efceb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:55.336814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.336780 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:55.338468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.338441 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-xks-ca" pod "67a76671-86e0-4a7e-afa6-6d0649efceb2" (UID: "67a76671-86e0-4a7e-afa6-6d0649efceb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:21:55.338625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.338608 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a76671-86e0-4a7e-afa6-6d0649efceb2-kube-api-access-dpz2g" pod "67a76671-86e0-4a7e-afa6-6d0649efceb2" (UID: "67a76671-86e0-4a7e-afa6-6d0649efceb2"). InnerVolumeSpecName "kube-api-access-dpz2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:55.438287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.438207 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a76671-86e0-4a7e-afa6-6d0649efceb2-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:55.438287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.438236 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpz2g\" (UniqueName: \"kubernetes.io/projected/67a76671-86e0-4a7e-afa6-6d0649efceb2-kube-api-access-dpz2g\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:21:55.687313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:55.687268 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:21:56.047403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:56.047369 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb"] Nov 01 20:21:56.095386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:56.095344 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b737cb6c-2sswb"] Nov 01 20:21:56.118822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:56.118790 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" Nov 01 20:21:56.118985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:56.118838 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd" event={"ID":"67a76671-86e0-4a7e-afa6-6d0649efceb2","Type":"ContainerDied","Data":"ee38bc776ba90588799dd215be80b610533ac7e91c2cc2a14be770944748f2b8"} Nov 01 20:21:56.118985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:56.118899 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee38bc776ba90588799dd215be80b610533ac7e91c2cc2a14be770944748f2b8" Nov 01 20:21:56.468971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:56.468939 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c7c6a6-d853-48e8-ba56-db17fff6f46b" path="/var/lib/kubelet/pods/b6c7c6a6-d853-48e8-ba56-db17fff6f46b/volumes" Nov 01 20:21:57.066623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:57.066586 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:21:58.390028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.389992 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8"] Nov 01 20:21:58.391662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.391633 5633 cpu_manager.go:414] "containerMap: removing container" podUID="67a76671-86e0-4a7e-afa6-6d0649efceb2" containerName="revoke-keys" Nov 01 20:21:58.391831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.391816 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a76671-86e0-4a7e-afa6-6d0649efceb2" containerName="revoke-keys" Nov 01 20:21:58.392195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.392175 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a76671-86e0-4a7e-afa6-6d0649efceb2" containerName="revoke-keys" Nov 01 20:21:58.411839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.411805 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.456416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.456375 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8"] Nov 01 20:21:58.486747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.486707 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:21:58.486919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.486862 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:21:58.486984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.486932 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:21:58.542304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.527745 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:21:58.564364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.552136 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.564895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.564854 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkfk\" (UniqueName: \"kubernetes.io/projected/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-kube-api-access-xgkfk\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.565153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.565133 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.679388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.679265 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.679792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.679763 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.680083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.680062 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkfk\" (UniqueName: \"kubernetes.io/projected/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-kube-api-access-xgkfk\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.681468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.681443 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.729769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.729643 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.767946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.767909 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkfk\" (UniqueName: \"kubernetes.io/projected/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-kube-api-access-xgkfk\") pod \"maas-api-revoke-sub-b6b243f7-kg8q8\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:58.899245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:58.897657 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:21:59.737356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:59.707470 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:21:59.901464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:59.897431 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 20:21:59.901464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:21:59.899184 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:22:00.275616 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:22:00.273499 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d840bb0_fbb2_4c28_922f_b4ae8f82e008.slice/crio-a1515392657bfa898a9b9fafdf8e5dfe3789d0385e82336074e2ffba8247b7bd WatchSource:0}: Error finding container a1515392657bfa898a9b9fafdf8e5dfe3789d0385e82336074e2ffba8247b7bd: Status 404 returned error can't find the container with id a1515392657bfa898a9b9fafdf8e5dfe3789d0385e82336074e2ffba8247b7bd Nov 01 20:22:00.284388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:00.284350 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8"] Nov 01 20:22:01.028191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:01.027987 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:01.380628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:01.380529 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" event={"ID":"4d840bb0-fbb2-4c28-922f-b4ae8f82e008","Type":"ContainerStarted","Data":"b52665af1805b8b3470456a1f7e65a5a09a352ab9987445e036eb0e23c6d8cd6"} Nov 01 20:22:01.380797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:01.380629 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" event={"ID":"4d840bb0-fbb2-4c28-922f-b4ae8f82e008","Type":"ContainerStarted","Data":"a1515392657bfa898a9b9fafdf8e5dfe3789d0385e82336074e2ffba8247b7bd"} Nov 01 20:22:01.512577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:01.511561 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" podStartSLOduration=3.511529398 podStartE2EDuration="3.511529398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:01.510894502 +0000 UTC m=+1129.185153758" watchObservedRunningTime="2026-11-01 20:22:01.511529398 +0000 UTC m=+1129.185788655" Nov 01 20:22:02.412885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:02.412841 5633 generic.go:356] "Generic (PLEG): container finished" podID="4d840bb0-fbb2-4c28-922f-b4ae8f82e008" containerID="b52665af1805b8b3470456a1f7e65a5a09a352ab9987445e036eb0e23c6d8cd6" exitCode=0 Nov 01 20:22:02.413466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:02.412907 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" event={"ID":"4d840bb0-fbb2-4c28-922f-b4ae8f82e008","Type":"ContainerDied","Data":"b52665af1805b8b3470456a1f7e65a5a09a352ab9987445e036eb0e23c6d8cd6"} Nov 01 20:22:03.619870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.619835 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:22:03.671551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.671519 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-xks-ca\") pod \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " Nov 01 20:22:03.671748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.671629 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-kube-api-access-xgkfk\" (UniqueName: \"kubernetes.io/projected/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-kube-api-access-xgkfk\") pod \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " Nov 01 20:22:03.671806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.671767 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-openshift-ca\") pod \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\" (UID: \"4d840bb0-fbb2-4c28-922f-b4ae8f82e008\") " Nov 01 20:22:03.672160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.672132 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-openshift-ca" pod "4d840bb0-fbb2-4c28-922f-b4ae8f82e008" (UID: "4d840bb0-fbb2-4c28-922f-b4ae8f82e008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:22:03.673828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.673778 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-xks-ca" pod "4d840bb0-fbb2-4c28-922f-b4ae8f82e008" (UID: "4d840bb0-fbb2-4c28-922f-b4ae8f82e008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:03.674011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.673991 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-kube-api-access-xgkfk" pod "4d840bb0-fbb2-4c28-922f-b4ae8f82e008" (UID: "4d840bb0-fbb2-4c28-922f-b4ae8f82e008"). InnerVolumeSpecName "kube-api-access-xgkfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:03.773679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.773641 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:03.773679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.773679 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgkfk\" (UniqueName: \"kubernetes.io/projected/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-kube-api-access-xgkfk\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:03.773881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:03.773695 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d840bb0-fbb2-4c28-922f-b4ae8f82e008-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:04.431665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:04.431631 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" Nov 01 20:22:04.431851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:04.431660 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8" event={"ID":"4d840bb0-fbb2-4c28-922f-b4ae8f82e008","Type":"ContainerDied","Data":"a1515392657bfa898a9b9fafdf8e5dfe3789d0385e82336074e2ffba8247b7bd"} Nov 01 20:22:04.431851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:04.431714 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1515392657bfa898a9b9fafdf8e5dfe3789d0385e82336074e2ffba8247b7bd" Nov 01 20:22:05.685541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:05.685502 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:06.419639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.419595 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" containerID="cri-o://7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1" gracePeriod=45 Nov 01 20:22:06.728314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.728278 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:22:06.815631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.815596 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-model-cache\") pod \"01a0661c-0fc6-43b8-8881-1177313e219b\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " Nov 01 20:22:06.815797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.815644 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a0661c-0fc6-43b8-8881-1177313e219b-kube-api-access-7lqcx\" (UniqueName: \"kubernetes.io/projected/01a0661c-0fc6-43b8-8881-1177313e219b-kube-api-access-7lqcx\") pod \"01a0661c-0fc6-43b8-8881-1177313e219b\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " Nov 01 20:22:06.815797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.815695 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-tmp-dir\") pod \"01a0661c-0fc6-43b8-8881-1177313e219b\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " Nov 01 20:22:06.815797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.815768 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-dshm\") pod \"01a0661c-0fc6-43b8-8881-1177313e219b\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " Nov 01 20:22:06.815979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.815890 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-model-cache" pod "01a0661c-0fc6-43b8-8881-1177313e219b" (UID: "01a0661c-0fc6-43b8-8881-1177313e219b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:06.815979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.815917 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a0661c-0fc6-43b8-8881-1177313e219b-tls-certs\" (UniqueName: \"kubernetes.io/secret/01a0661c-0fc6-43b8-8881-1177313e219b-tls-certs\") pod \"01a0661c-0fc6-43b8-8881-1177313e219b\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " Nov 01 20:22:06.816074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.816008 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-tmp-dir" pod "01a0661c-0fc6-43b8-8881-1177313e219b" (UID: "01a0661c-0fc6-43b8-8881-1177313e219b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:06.816074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.816058 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-home\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-home\") pod \"01a0661c-0fc6-43b8-8881-1177313e219b\" (UID: \"01a0661c-0fc6-43b8-8881-1177313e219b\") " Nov 01 20:22:06.816348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.816309 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-home" pod "01a0661c-0fc6-43b8-8881-1177313e219b" (UID: "01a0661c-0fc6-43b8-8881-1177313e219b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:06.816793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.816774 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:06.816860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.816801 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:06.816860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.816816 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:06.818293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.818275 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a0661c-0fc6-43b8-8881-1177313e219b-kube-api-access-7lqcx" pod "01a0661c-0fc6-43b8-8881-1177313e219b" (UID: "01a0661c-0fc6-43b8-8881-1177313e219b"). InnerVolumeSpecName "kube-api-access-7lqcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:06.818482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.818467 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-dshm" pod "01a0661c-0fc6-43b8-8881-1177313e219b" (UID: "01a0661c-0fc6-43b8-8881-1177313e219b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:06.818544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.818515 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a0661c-0fc6-43b8-8881-1177313e219b-tls-certs" pod "01a0661c-0fc6-43b8-8881-1177313e219b" (UID: "01a0661c-0fc6-43b8-8881-1177313e219b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:06.918077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.918042 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lqcx\" (UniqueName: \"kubernetes.io/projected/01a0661c-0fc6-43b8-8881-1177313e219b-kube-api-access-7lqcx\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:06.918077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.918072 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a0661c-0fc6-43b8-8881-1177313e219b-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:06.918077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:06.918083 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a0661c-0fc6-43b8-8881-1177313e219b-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:07.050490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.050454 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:07.458843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.458758 5633 generic.go:356] "Generic (PLEG): container finished" podID="01a0661c-0fc6-43b8-8881-1177313e219b" containerID="7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1" exitCode=0 Nov 01 20:22:07.458843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.458808 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" Nov 01 20:22:07.458843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.458827 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" event={"ID":"01a0661c-0fc6-43b8-8881-1177313e219b","Type":"ContainerDied","Data":"7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1"} Nov 01 20:22:07.459114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.458876 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf" event={"ID":"01a0661c-0fc6-43b8-8881-1177313e219b","Type":"ContainerDied","Data":"c71e533b136d88734834306358d3cfdfe86d03d19c5ef4b401ecc81725c80b28"} Nov 01 20:22:07.459114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.458895 5633 scope.go:122] "RemoveContainer" containerID="7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1" Nov 01 20:22:07.473832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.473810 5633 scope.go:122] "RemoveContainer" containerID="7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1" Nov 01 20:22:07.474204 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:22:07.474180 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1\": container with ID starting with 7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1 not found: ID does not exist" containerID="7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1" Nov 01 20:22:07.474351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.474209 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1"} err="failed to get container status \"7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1\": rpc error: code = NotFound desc = could not find container \"7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1\": container with ID starting with 7326198d073887e3460eacd69012ba85fbe0a5993a9382b9a480d53a15fdfab1 not found: ID does not exist" Nov 01 20:22:07.527128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.527092 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:07.623636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.623600 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf"] Nov 01 20:22:07.673124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:07.673090 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ad7edb/test-model-ad7edb-kserve-b6966ccb9-ltbqf"] Nov 01 20:22:08.467979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:08.467948 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" path="/var/lib/kubelet/pods/01a0661c-0fc6-43b8-8881-1177313e219b/volumes" Nov 01 20:22:09.478961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:09.478924 5633 generic.go:356] "Generic (PLEG): container finished" podID="599e829d-0f1c-496e-bf40-f476c03bb904" containerID="93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827" exitCode=0 Nov 01 20:22:09.479424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:09.478998 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" event={"ID":"599e829d-0f1c-496e-bf40-f476c03bb904","Type":"ContainerDied","Data":"93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827"} Nov 01 20:22:09.479475 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:09.479464 5633 scope.go:122] "RemoveContainer" containerID="93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827" Nov 01 20:22:09.481123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:09.481097 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:22:09.481194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:09.481171 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:22:09.481229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:09.481206 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:22:10.410973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:10.410935 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:10.522068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:10.521996 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" event={"ID":"599e829d-0f1c-496e-bf40-f476c03bb904","Type":"ContainerStarted","Data":"51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f"} Nov 01 20:22:11.050733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:11.050694 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4"] Nov 01 20:22:11.054718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:11.054695 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87a329-rwlx4"] Nov 01 20:22:11.108370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:11.107701 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:22:11.111183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:11.111140 5633 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:22:11.111436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:11.111208 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 20:22:12.110361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:12.110299 5633 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:22:12.110806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:12.110387 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 20:22:12.467698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:12.467668 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ec2944-b62c-467b-8f03-7bccc4abc4fc" path="/var/lib/kubelet/pods/c3ec2944-b62c-467b-8f03-7bccc4abc4fc/volumes" Nov 01 20:22:12.962041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:12.962016 5633 scope.go:122] "RemoveContainer" containerID="803e11dbb7a8bdda1d9dfc3e316e7a3a6948860fb0bd725f59ca5f22e1be428c" Nov 01 20:22:12.973126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:12.973100 5633 scope.go:122] "RemoveContainer" containerID="fddcfd3fa3f3f3c686632db4311408deefb67bd6194884e2b0f9de684bb476b0" Nov 01 20:22:13.012389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:13.012310 5633 scope.go:122] "RemoveContainer" containerID="527852c529d046bc69db8971b7baccf65500ad75e91d420f5ef03da4080fad9f" Nov 01 20:22:13.110508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:13.108610 5633 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:22:13.110508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:13.108673 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 20:22:13.231749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:13.231662 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:13.452894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:13.452857 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2"] Nov 01 20:22:13.713449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:13.712919 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:13.845124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:13.845087 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:14.097863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:14.097764 5633 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:22:14.098174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:14.098132 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 20:22:14.262439 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:14.262402 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:22:15.102357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.100772 5633 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:22:15.102357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.100859 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 20:22:15.536370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.536290 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:15.945993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.945900 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5"] Nov 01 20:22:15.946870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.946849 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4d840bb0-fbb2-4c28-922f-b4ae8f82e008" containerName="revoke-keys" Nov 01 20:22:15.946918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.946874 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d840bb0-fbb2-4c28-922f-b4ae8f82e008" containerName="revoke-keys" Nov 01 20:22:15.946918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.946897 5633 cpu_manager.go:414] "containerMap: removing container" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" Nov 01 20:22:15.946918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.946904 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" Nov 01 20:22:15.947084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.947072 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d840bb0-fbb2-4c28-922f-b4ae8f82e008" containerName="revoke-keys" Nov 01 20:22:15.947119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.947089 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a0661c-0fc6-43b8-8881-1177313e219b" containerName="main" Nov 01 20:22:15.952526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.952501 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:15.984820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.984785 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:15.984990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.984929 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbs2k\" (UniqueName: \"kubernetes.io/projected/d7761247-0c6c-43f3-aaa2-ecd182349257-kube-api-access-sbs2k\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:15.984990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:15.984978 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.004926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.004895 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5"] Nov 01 20:22:16.049519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.049478 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:22:16.049519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.049482 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:22:16.053750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.053728 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:22:16.053750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.053740 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:22:16.086804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.086765 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbs2k\" (UniqueName: \"kubernetes.io/projected/d7761247-0c6c-43f3-aaa2-ecd182349257-kube-api-access-sbs2k\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.086978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.086831 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.086978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.086918 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.087595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.087576 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.089356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.089318 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.097309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.097267 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:22:16.098891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.098829 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:22:16.144654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.144620 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbs2k\" (UniqueName: \"kubernetes.io/projected/d7761247-0c6c-43f3-aaa2-ecd182349257-kube-api-access-sbs2k\") pod \"maas-api-revoke-sub-79cccee6-7ztt5\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.158274 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:22:16.158230 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:22:16.268482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.268445 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:16.648321 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:22:16.648288 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7761247_0c6c_43f3_aaa2_ecd182349257.slice/crio-ea4399173ba7a44ccaa12887a45fa2414a26edcb9883ecc17b5a1b07a6f075f3 WatchSource:0}: Error finding container ea4399173ba7a44ccaa12887a45fa2414a26edcb9883ecc17b5a1b07a6f075f3: Status 404 returned error can't find the container with id ea4399173ba7a44ccaa12887a45fa2414a26edcb9883ecc17b5a1b07a6f075f3 Nov 01 20:22:16.648784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.648480 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5"] Nov 01 20:22:16.677946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:16.677914 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" event={"ID":"d7761247-0c6c-43f3-aaa2-ecd182349257","Type":"ContainerStarted","Data":"ea4399173ba7a44ccaa12887a45fa2414a26edcb9883ecc17b5a1b07a6f075f3"} Nov 01 20:22:17.687399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:17.687361 5633 generic.go:356] "Generic (PLEG): container finished" podID="d7761247-0c6c-43f3-aaa2-ecd182349257" containerID="c8582737cede57be225c1567b883c5bd566b739aadeccb62c24c0e5a00a26bf2" exitCode=0 Nov 01 20:22:17.687802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:17.687370 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" event={"ID":"d7761247-0c6c-43f3-aaa2-ecd182349257","Type":"ContainerDied","Data":"c8582737cede57be225c1567b883c5bd566b739aadeccb62c24c0e5a00a26bf2"} Nov 01 20:22:18.453562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.453515 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" podUID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" containerName="payload-pre-processing" containerID="cri-o://968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c" gracePeriod=25 Nov 01 20:22:18.866230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.866203 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:18.926174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.926136 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-xks-ca\") pod \"d7761247-0c6c-43f3-aaa2-ecd182349257\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " Nov 01 20:22:18.926174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.926181 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7761247-0c6c-43f3-aaa2-ecd182349257-kube-api-access-sbs2k\" (UniqueName: \"kubernetes.io/projected/d7761247-0c6c-43f3-aaa2-ecd182349257-kube-api-access-sbs2k\") pod \"d7761247-0c6c-43f3-aaa2-ecd182349257\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " Nov 01 20:22:18.926429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.926220 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-openshift-ca\") pod \"d7761247-0c6c-43f3-aaa2-ecd182349257\" (UID: \"d7761247-0c6c-43f3-aaa2-ecd182349257\") " Nov 01 20:22:18.926664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.926643 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-openshift-ca" pod "d7761247-0c6c-43f3-aaa2-ecd182349257" (UID: "d7761247-0c6c-43f3-aaa2-ecd182349257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:22:18.928525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.928502 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7761247-0c6c-43f3-aaa2-ecd182349257-kube-api-access-sbs2k" pod "d7761247-0c6c-43f3-aaa2-ecd182349257" (UID: "d7761247-0c6c-43f3-aaa2-ecd182349257"). InnerVolumeSpecName "kube-api-access-sbs2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:18.928664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:18.928543 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-xks-ca" pod "d7761247-0c6c-43f3-aaa2-ecd182349257" (UID: "d7761247-0c6c-43f3-aaa2-ecd182349257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:19.002266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:19.002169 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:19.027887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:19.027854 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:19.027887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:19.027885 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbs2k\" (UniqueName: \"kubernetes.io/projected/d7761247-0c6c-43f3-aaa2-ecd182349257-kube-api-access-sbs2k\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:19.027887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:19.027895 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7761247-0c6c-43f3-aaa2-ecd182349257-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:19.705491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:19.705464 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" Nov 01 20:22:19.705678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:19.705493 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5" event={"ID":"d7761247-0c6c-43f3-aaa2-ecd182349257","Type":"ContainerDied","Data":"ea4399173ba7a44ccaa12887a45fa2414a26edcb9883ecc17b5a1b07a6f075f3"} Nov 01 20:22:19.705678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:19.705549 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4399173ba7a44ccaa12887a45fa2414a26edcb9883ecc17b5a1b07a6f075f3" Nov 01 20:22:20.505599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:20.505563 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:23.832294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:23.832255 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:25.202082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:25.202046 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:25.237143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:25.237091 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" podUID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:22:26.668901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.668860 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5"] Nov 01 20:22:26.669579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.669561 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d7761247-0c6c-43f3-aaa2-ecd182349257" containerName="revoke-keys" Nov 01 20:22:26.669654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.669581 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7761247-0c6c-43f3-aaa2-ecd182349257" containerName="revoke-keys" Nov 01 20:22:26.669701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.669692 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7761247-0c6c-43f3-aaa2-ecd182349257" containerName="revoke-keys" Nov 01 20:22:26.673689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.673671 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.729239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.729202 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5"] Nov 01 20:22:26.773363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.773307 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:22:26.773517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.773384 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:22:26.775146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.775131 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:22:26.775146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.775136 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:22:26.831553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.831521 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.831733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.831572 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2q5\" (UniqueName: \"kubernetes.io/projected/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-kube-api-access-2z2q5\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.831778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.831739 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.933407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.933292 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.933563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.933426 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.933563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.933464 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2q5\" (UniqueName: \"kubernetes.io/projected/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-kube-api-access-2z2q5\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.934149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.934120 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.936009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.935975 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.990134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.990096 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2q5\" (UniqueName: \"kubernetes.io/projected/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-kube-api-access-2z2q5\") pod \"maas-api-revoke-sub-480dc2c4-bv6h5\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:26.991978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:26.991953 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:27.377191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:27.377159 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5"] Nov 01 20:22:27.770589 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:27.770556 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" event={"ID":"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa","Type":"ContainerStarted","Data":"ab2bf75ae22a399343345ba6625b8267ef2a185ba941b9261cb4fc1b22bc1006"} Nov 01 20:22:27.770974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:27.770614 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" event={"ID":"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa","Type":"ContainerStarted","Data":"3cd1c6d8cc7387f23518e87152a9a8c2803c925a552489e2b9c0e35454c2b671"} Nov 01 20:22:27.884137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:27.884068 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" podStartSLOduration=1.884052301 podStartE2EDuration="1.884052301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:27.882034603 +0000 UTC m=+1155.556293859" watchObservedRunningTime="2026-11-01 20:22:27.884052301 +0000 UTC m=+1155.558311556" Nov 01 20:22:28.265349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:28.265302 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:22:28.786744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:28.786709 5633 generic.go:356] "Generic (PLEG): container finished" podID="ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" containerID="ab2bf75ae22a399343345ba6625b8267ef2a185ba941b9261cb4fc1b22bc1006" exitCode=0 Nov 01 20:22:28.787136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:28.786798 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" event={"ID":"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa","Type":"ContainerDied","Data":"ab2bf75ae22a399343345ba6625b8267ef2a185ba941b9261cb4fc1b22bc1006"} Nov 01 20:22:29.399798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.397226 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl"] Nov 01 20:22:29.405356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.403916 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.483971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.483921 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl"] Nov 01 20:22:29.491362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.491123 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:29.523690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.523651 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"openshift-service-ca.crt\"" Nov 01 20:22:29.523876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.523795 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"distinct-2-w0-d204f4-kserve-self-signed-certs\"" Nov 01 20:22:29.523876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.523849 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"default-dockercfg-qdjvd\"" Nov 01 20:22:29.524020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.523942 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"kube-root-ca.crt\"" Nov 01 20:22:29.596480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.596427 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-tmp-dir\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.608835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.596857 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-dshm\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.608835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.607781 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/123292cc-88ae-449c-80a1-793b135d8a4a-tls-certs\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.608835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.607897 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrml\" (UniqueName: \"kubernetes.io/projected/123292cc-88ae-449c-80a1-793b135d8a4a-kube-api-access-mjrml\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.608835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.608034 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-model-cache\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.608835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.608086 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-home\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.710560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.710473 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-dshm\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.711320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.711294 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/123292cc-88ae-449c-80a1-793b135d8a4a-tls-certs\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.711983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.711962 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrml\" (UniqueName: \"kubernetes.io/projected/123292cc-88ae-449c-80a1-793b135d8a4a-kube-api-access-mjrml\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.712586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.712563 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-model-cache\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.713566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.713549 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-home\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.713835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.713819 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-tmp-dir\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.723806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.713024 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-model-cache\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.723806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.714221 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-home\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.723806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.714468 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-tmp-dir\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.723806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.714632 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-dshm\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.736367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.735050 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/123292cc-88ae-449c-80a1-793b135d8a4a-tls-certs\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:29.773494 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:29.773459 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrml\" (UniqueName: \"kubernetes.io/projected/123292cc-88ae-449c-80a1-793b135d8a4a-kube-api-access-mjrml\") pod \"distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:30.039635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:30.039594 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:30.229204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:30.229169 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:31.199357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.188712 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:31.199357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.189194 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl"] Nov 01 20:22:31.305020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.303714 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-kube-api-access-2z2q5\" (UniqueName: \"kubernetes.io/projected/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-kube-api-access-2z2q5\") pod \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " Nov 01 20:22:31.305640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.305613 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-xks-ca\") pod \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " Nov 01 20:22:31.305848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.305831 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-openshift-ca\") pod \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\" (UID: \"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa\") " Nov 01 20:22:31.308233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.308208 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-openshift-ca" pod "ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" (UID: "ba3a31ae-dca4-4000-99c7-ca1a1cb997aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:22:31.313496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.313468 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-kube-api-access-2z2q5" pod "ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" (UID: "ba3a31ae-dca4-4000-99c7-ca1a1cb997aa"). InnerVolumeSpecName "kube-api-access-2z2q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:31.384302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.367695 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-xks-ca" pod "ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" (UID: "ba3a31ae-dca4-4000-99c7-ca1a1cb997aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:31.450372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.442555 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z2q5\" (UniqueName: \"kubernetes.io/projected/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-kube-api-access-2z2q5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:31.450372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.442591 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:31.450372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.442609 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:31.938247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.938211 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" event={"ID":"ba3a31ae-dca4-4000-99c7-ca1a1cb997aa","Type":"ContainerDied","Data":"3cd1c6d8cc7387f23518e87152a9a8c2803c925a552489e2b9c0e35454c2b671"} Nov 01 20:22:31.938473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.938267 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd1c6d8cc7387f23518e87152a9a8c2803c925a552489e2b9c0e35454c2b671" Nov 01 20:22:31.938473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.938222 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5" Nov 01 20:22:31.940217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.940188 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" event={"ID":"123292cc-88ae-449c-80a1-793b135d8a4a","Type":"ContainerStarted","Data":"97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a"} Nov 01 20:22:31.940388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:31.940231 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" event={"ID":"123292cc-88ae-449c-80a1-793b135d8a4a","Type":"ContainerStarted","Data":"f35615ab5a0af43cf94f64c67bbcade190df70a9330cc36e6778ab1b2dfd6947"} Nov 01 20:22:32.046968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:32.046938 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk"] Nov 01 20:22:32.051277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:32.051243 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60f273a-vp8wk"] Nov 01 20:22:32.061607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:32.061579 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:32.093565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:32.093511 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podStartSLOduration=3.093495395 podStartE2EDuration="3.093495395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:32.091662384 +0000 UTC m=+1159.765921637" watchObservedRunningTime="2026-11-01 20:22:32.093495395 +0000 UTC m=+1159.767754687" Nov 01 20:22:32.467520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:32.467487 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f848e91c-1ab9-46d9-9ca9-724557034e45" path="/var/lib/kubelet/pods/f848e91c-1ab9-46d9-9ca9-724557034e45/volumes" Nov 01 20:22:32.800963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:32.800842 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:33.328158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:33.328119 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:33.968157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:33.968122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:34.144373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:34.144291 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:34.278859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:34.278766 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:34.495093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:34.495065 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:34.659755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:34.659660 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:34.923075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:34.922988 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:35.238651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:35.238613 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" podUID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:22:35.496317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:35.496276 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:35.681569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:35.681532 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:35.807660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:35.807571 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:37.786117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:37.786080 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:38.333199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.333166 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw"] Nov 01 20:22:38.333835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.333819 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" containerName="revoke-keys" Nov 01 20:22:38.333900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.333837 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" containerName="revoke-keys" Nov 01 20:22:38.333938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.333928 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" containerName="revoke-keys" Nov 01 20:22:38.338967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.338944 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.397022 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.396985 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw"] Nov 01 20:22:38.437157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.437121 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\"" Nov 01 20:22:38.437157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.437127 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-4bc05b0bda98\"" Nov 01 20:22:38.437430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.437169 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\"" Nov 01 20:22:38.437430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.437138 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\"" Nov 01 20:22:38.443842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.443807 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-t8dl8\"" Nov 01 20:22:38.443842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.443815 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\"" Nov 01 20:22:38.455536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.455501 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.455703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.455634 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.455838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.455820 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.455891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.455868 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9vw\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-kube-api-access-jm9vw\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.557117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.557074 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.557358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.557165 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9vw\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-kube-api-access-jm9vw\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.557358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.557239 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.557501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.557362 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.558050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.558028 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.558257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.558225 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.559899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.559877 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.613624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.613550 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9vw\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-kube-api-access-jm9vw\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.654916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.654881 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:38.756129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.755776 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:22:38.861423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.861114 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-kube-api-access-jj2jk\" (UniqueName: \"kubernetes.io/projected/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-kube-api-access-jj2jk\") pod \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " Nov 01 20:22:38.861423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.861216 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-plugins-config-volume\") pod \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\" (UID: \"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae\") " Nov 01 20:22:38.861898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.861604 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-plugins-config-volume" pod "fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" (UID: "fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:22:38.862086 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.862065 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:38.863560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.863532 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-kube-api-access-jj2jk" pod "fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" (UID: "fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae"). InnerVolumeSpecName "kube-api-access-jj2jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:38.963034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:38.963007 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2jk\" (UniqueName: \"kubernetes.io/projected/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae-kube-api-access-jj2jk\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:39.005631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.005592 5633 generic.go:356] "Generic (PLEG): container finished" podID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" containerID="968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c" exitCode=0 Nov 01 20:22:39.005828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.005638 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" Nov 01 20:22:39.005828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.005675 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" event={"ID":"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae","Type":"ContainerDied","Data":"968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c"} Nov 01 20:22:39.005828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.005728 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2" event={"ID":"fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae","Type":"ContainerDied","Data":"91f119e72726fda9b884d5b5efa9b62cdce3456576c35bc8e60b228affd7ace0"} Nov 01 20:22:39.005828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.005756 5633 scope.go:122] "RemoveContainer" containerID="968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c" Nov 01 20:22:39.016938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.016918 5633 scope.go:122] "RemoveContainer" containerID="968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c" Nov 01 20:22:39.017271 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:22:39.017244 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c\": container with ID starting with 968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c not found: ID does not exist" containerID="968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c" Nov 01 20:22:39.017346 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.017272 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c"} err="failed to get container status \"968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c\": rpc error: code = NotFound desc = could not find container \"968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c\": container with ID starting with 968ab73a623fad32cca2520a240bb3d15e8a9e8332380c07d88b50a3607f142c not found: ID does not exist" Nov 01 20:22:39.147855 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:22:39.147823 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe142b44_da56_4c21_b797_944c98c0362a.slice/crio-b546ad3acf4f96df33cfcf58f9649a876272a4c1c8ddf4727cb33e2f8d2e41b6 WatchSource:0}: Error finding container b546ad3acf4f96df33cfcf58f9649a876272a4c1c8ddf4727cb33e2f8d2e41b6: Status 404 returned error can't find the container with id b546ad3acf4f96df33cfcf58f9649a876272a4c1c8ddf4727cb33e2f8d2e41b6 Nov 01 20:22:39.148476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.148321 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw"] Nov 01 20:22:39.170075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.169998 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2"] Nov 01 20:22:39.175832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:39.175801 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ad7edb-b76cf98ff-qprz2"] Nov 01 20:22:40.015271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.015241 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" event={"ID":"fe142b44-da56-4c21-b797-944c98c0362a","Type":"ContainerStarted","Data":"4962d4de0e7eabea7a370e985ab76aa066ab8003b763dd6f9e26ac3a14d3aa0a"} Nov 01 20:22:40.015792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.015309 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:40.015792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.015344 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" event={"ID":"fe142b44-da56-4c21-b797-944c98c0362a","Type":"ContainerStarted","Data":"b546ad3acf4f96df33cfcf58f9649a876272a4c1c8ddf4727cb33e2f8d2e41b6"} Nov 01 20:22:40.060926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.060888 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:40.073719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.073690 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:40.131966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.131910 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" podStartSLOduration=2.131894093 podStartE2EDuration="2.131894093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:40.128020325 +0000 UTC m=+1167.802279581" watchObservedRunningTime="2026-11-01 20:22:40.131894093 +0000 UTC m=+1167.806153362" Nov 01 20:22:40.187242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.187206 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:40.467600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:40.467569 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" path="/var/lib/kubelet/pods/fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae/volumes" Nov 01 20:22:41.039283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:41.039239 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:42.443154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:42.443116 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:43.104948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:43.102528 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62"] Nov 01 20:22:43.127919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:43.127877 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7882123d-srh62"] Nov 01 20:22:44.032546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:44.032489 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 01 20:22:44.591441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:44.589496 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e830b283-0b3d-49de-97c2-9a59499bb1de" path="/var/lib/kubelet/pods/e830b283-0b3d-49de-97c2-9a59499bb1de/volumes" Nov 01 20:22:44.707972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:44.707912 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:22:44.762546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:44.762502 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:22:45.035199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:45.035152 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:22:45.057360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:45.036364 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:45.773359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:45.772693 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 01 20:22:45.785371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:45.773636 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:22:46.088409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:46.087632 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 01 20:22:46.502803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:46.502759 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:46.700385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:46.699606 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:22:46.702080 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:22:46.702050 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:22:46.703253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:46.703174 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:22:50.716201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:50.716162 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:51.040413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:51.040241 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:22:52.514663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:52.514622 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:53.619735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.619707 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt"] Nov 01 20:22:53.620361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.620317 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" containerName="payload-pre-processing" Nov 01 20:22:53.620361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.620346 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" containerName="payload-pre-processing" Nov 01 20:22:53.620470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.620460 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0baf5d-cbf6-444e-ab7a-972d89d1a3ae" containerName="payload-pre-processing" Nov 01 20:22:53.625475 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.625457 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.680158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.680120 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt"] Nov 01 20:22:53.704397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.704308 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.704617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.704439 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.704617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.704568 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxp8\" (UniqueName: \"kubernetes.io/projected/b93f0813-21c0-4426-8f75-ffdcdee9efdd-kube-api-access-tgxp8\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.723228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.723196 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:22:53.723463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.723449 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:22:53.723519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.723452 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:22:53.723557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.723530 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:22:53.805818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.805783 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxp8\" (UniqueName: \"kubernetes.io/projected/b93f0813-21c0-4426-8f75-ffdcdee9efdd-kube-api-access-tgxp8\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.805995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.805975 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.806049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.806018 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.806728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.806703 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.808584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.808563 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.862278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.862240 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxp8\" (UniqueName: \"kubernetes.io/projected/b93f0813-21c0-4426-8f75-ffdcdee9efdd-kube-api-access-tgxp8\") pod \"maas-api-revoke-sub-9f42b71e-trmkt\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:53.940365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:53.940246 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:54.332040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:54.329077 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt"] Nov 01 20:22:54.629546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:54.629497 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" event={"ID":"b93f0813-21c0-4426-8f75-ffdcdee9efdd","Type":"ContainerStarted","Data":"108106c0a4cffd1a45e91728d205646e9f24904ec2cae786ecbbac2f3ebdb963"} Nov 01 20:22:54.629984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:54.629603 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" event={"ID":"b93f0813-21c0-4426-8f75-ffdcdee9efdd","Type":"ContainerStarted","Data":"1afe0936bb6b26b27f2b928ace852f410c26ceab87da8bf2c64c7596655f4ddd"} Nov 01 20:22:54.746444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:54.746393 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" podStartSLOduration=1.746379406 podStartE2EDuration="1.746379406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:54.743533988 +0000 UTC m=+1182.417793243" watchObservedRunningTime="2026-11-01 20:22:54.746379406 +0000 UTC m=+1182.420638701" Nov 01 20:22:55.639681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:55.639646 5633 generic.go:356] "Generic (PLEG): container finished" podID="b93f0813-21c0-4426-8f75-ffdcdee9efdd" containerID="108106c0a4cffd1a45e91728d205646e9f24904ec2cae786ecbbac2f3ebdb963" exitCode=0 Nov 01 20:22:55.640051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:55.639699 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" event={"ID":"b93f0813-21c0-4426-8f75-ffdcdee9efdd","Type":"ContainerDied","Data":"108106c0a4cffd1a45e91728d205646e9f24904ec2cae786ecbbac2f3ebdb963"} Nov 01 20:22:56.809009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.808975 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:56.853319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.853285 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-xks-ca\") pod \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " Nov 01 20:22:56.853550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.853437 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93f0813-21c0-4426-8f75-ffdcdee9efdd-kube-api-access-tgxp8\" (UniqueName: \"kubernetes.io/projected/b93f0813-21c0-4426-8f75-ffdcdee9efdd-kube-api-access-tgxp8\") pod \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " Nov 01 20:22:56.853550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.853494 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-openshift-ca\") pod \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\" (UID: \"b93f0813-21c0-4426-8f75-ffdcdee9efdd\") " Nov 01 20:22:56.853876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.853846 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-openshift-ca" pod "b93f0813-21c0-4426-8f75-ffdcdee9efdd" (UID: "b93f0813-21c0-4426-8f75-ffdcdee9efdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:22:56.855725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.855690 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-xks-ca" pod "b93f0813-21c0-4426-8f75-ffdcdee9efdd" (UID: "b93f0813-21c0-4426-8f75-ffdcdee9efdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:56.855725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.855703 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93f0813-21c0-4426-8f75-ffdcdee9efdd-kube-api-access-tgxp8" pod "b93f0813-21c0-4426-8f75-ffdcdee9efdd" (UID: "b93f0813-21c0-4426-8f75-ffdcdee9efdd"). InnerVolumeSpecName "kube-api-access-tgxp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:56.955565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.955467 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgxp8\" (UniqueName: \"kubernetes.io/projected/b93f0813-21c0-4426-8f75-ffdcdee9efdd-kube-api-access-tgxp8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:56.955565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.955498 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:56.955565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:56.955509 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93f0813-21c0-4426-8f75-ffdcdee9efdd-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:57.188386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:57.188312 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:57.657354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:57.657297 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" Nov 01 20:22:57.657354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:57.657341 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt" event={"ID":"b93f0813-21c0-4426-8f75-ffdcdee9efdd","Type":"ContainerDied","Data":"1afe0936bb6b26b27f2b928ace852f410c26ceab87da8bf2c64c7596655f4ddd"} Nov 01 20:22:57.657565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:57.657394 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1afe0936bb6b26b27f2b928ace852f410c26ceab87da8bf2c64c7596655f4ddd" Nov 01 20:22:58.725045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:58.725008 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:22:59.133189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:59.133097 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:22:59.253870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:22:59.253820 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:00.800364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:00.798090 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:01.416121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:01.416069 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:01.426012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:01.425979 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv"] Nov 01 20:23:01.426553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:01.426518 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" containerID="cri-o://2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba" gracePeriod=30 Nov 01 20:23:01.732993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:01.732945 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:02.028124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:02.028028 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:02.549522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:02.549486 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:04.425470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.425430 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc"] Nov 01 20:23:04.426043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.426029 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b93f0813-21c0-4426-8f75-ffdcdee9efdd" containerName="revoke-keys" Nov 01 20:23:04.426091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.426043 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93f0813-21c0-4426-8f75-ffdcdee9efdd" containerName="revoke-keys" Nov 01 20:23:04.426163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.426154 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93f0813-21c0-4426-8f75-ffdcdee9efdd" containerName="revoke-keys" Nov 01 20:23:04.430093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.430069 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.482292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.482238 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:23:04.495553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.495523 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nftst\" (UniqueName: \"kubernetes.io/projected/d064b2fd-3070-4712-9f0a-471b3d1e837c-kube-api-access-nftst\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.495855 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.495832 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.495966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.495949 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.531951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.531913 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:23:04.532136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.531960 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:23:04.532136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.531959 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:23:04.601771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.601727 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nftst\" (UniqueName: \"kubernetes.io/projected/d064b2fd-3070-4712-9f0a-471b3d1e837c-kube-api-access-nftst\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.602154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.602118 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.602324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.602223 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.603065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.603041 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.604922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.604897 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.658765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.658730 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nftst\" (UniqueName: \"kubernetes.io/projected/d064b2fd-3070-4712-9f0a-471b3d1e837c-kube-api-access-nftst\") pod \"maas-api-revoke-sub-e38f9c29-dm7nc\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:04.744479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:04.744440 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:06.720465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.720430 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:23:06.835508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835472 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113e6f55-ced3-4c66-b51d-83f8bc007233-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113e6f55-ced3-4c66-b51d-83f8bc007233-istiod-ca-cert\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.835729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835550 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-kube-api-access-5dwsz\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-kube-api-access-5dwsz\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.835729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835603 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-socket\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.835843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835745 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-podinfo\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.835843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835804 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-data\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.835936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835918 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-token\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-token\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.835984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835952 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-socket" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:06.836040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.835983 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113e6f55-ced3-4c66-b51d-83f8bc007233-istiod-ca-cert" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:06.836127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836109 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-certs\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.836196 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836180 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-envoy\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.836259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836227 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-credential-socket\") pod \"113e6f55-ced3-4c66-b51d-83f8bc007233\" (UID: \"113e6f55-ced3-4c66-b51d-83f8bc007233\") " Nov 01 20:23:06.836540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836183 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-data" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:06.836682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836590 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-certs" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:06.837012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836937 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.837012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836967 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113e6f55-ced3-4c66-b51d-83f8bc007233-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.837012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836983 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.837012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836982 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-credential-socket" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:06.837012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.836999 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.838228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.838201 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-kube-api-access-5dwsz" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "kube-api-access-5dwsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:06.838550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.838534 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-envoy" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:06.838646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.838624 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-podinfo" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:23:06.838722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.838708 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-token" pod "113e6f55-ced3-4c66-b51d-83f8bc007233" (UID: "113e6f55-ced3-4c66-b51d-83f8bc007233"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:06.872902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.872866 5633 generic.go:356] "Generic (PLEG): container finished" podID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerID="2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba" exitCode=0 Nov 01 20:23:06.873447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.872983 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" Nov 01 20:23:06.873447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.872981 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" event={"ID":"113e6f55-ced3-4c66-b51d-83f8bc007233","Type":"ContainerDied","Data":"2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba"} Nov 01 20:23:06.873447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.873039 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv" event={"ID":"113e6f55-ced3-4c66-b51d-83f8bc007233","Type":"ContainerDied","Data":"1f50d8176c0204ddc161d887975240f4411800bfd47bd32e6c6e6cc22d63a5c7"} Nov 01 20:23:06.873447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.873062 5633 scope.go:122] "RemoveContainer" containerID="2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba" Nov 01 20:23:06.885971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.885945 5633 scope.go:122] "RemoveContainer" containerID="bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569" Nov 01 20:23:06.898216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.898186 5633 scope.go:122] "RemoveContainer" containerID="2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba" Nov 01 20:23:06.898639 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:23:06.898617 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba\": container with ID starting with 2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba not found: ID does not exist" containerID="2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba" Nov 01 20:23:06.898747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.898654 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba"} err="failed to get container status \"2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba\": rpc error: code = NotFound desc = could not find container \"2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba\": container with ID starting with 2af8f5c710cbe71e938b51db4869a2eb933ec488d16634a88f83dd995c65fdba not found: ID does not exist" Nov 01 20:23:06.898747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.898685 5633 scope.go:122] "RemoveContainer" containerID="bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569" Nov 01 20:23:06.899036 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:23:06.899017 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569\": container with ID starting with bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569 not found: ID does not exist" containerID="bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569" Nov 01 20:23:06.899089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.899044 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569"} err="failed to get container status \"bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569\": rpc error: code = NotFound desc = could not find container \"bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569\": container with ID starting with bd755a61c2619a0cf0336aad01f18868b75c2b990ddc6f0d8a5ebe415a22d569 not found: ID does not exist" Nov 01 20:23:06.938222 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.938185 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.938222 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.938223 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113e6f55-ced3-4c66-b51d-83f8bc007233-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.938497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.938241 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwsz\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-kube-api-access-5dwsz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.938497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.938252 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:06.938497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:06.938261 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113e6f55-ced3-4c66-b51d-83f8bc007233-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:07.001707 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:07.001551 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv"] Nov 01 20:23:07.008518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:07.008482 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-7t7hv"] Nov 01 20:23:07.503225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:07.503190 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:07.831233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:07.831135 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:08.467952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:08.467919 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" path="/var/lib/kubelet/pods/113e6f55-ced3-4c66-b51d-83f8bc007233/volumes" Nov 01 20:23:09.265717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:09.265676 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:09.388058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:09.388024 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:10.728703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.728668 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7"] Nov 01 20:23:10.729322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.729308 5633 cpu_manager.go:414] "containerMap: removing container" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" Nov 01 20:23:10.729406 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.729323 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" Nov 01 20:23:10.729406 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.729371 5633 cpu_manager.go:414] "containerMap: removing container" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" Nov 01 20:23:10.729406 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.729377 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" Nov 01 20:23:10.729500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.729488 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" Nov 01 20:23:10.734480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.734464 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.798056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.798019 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dslqs\" (UniqueName: \"kubernetes.io/projected/4cbbfa87-4335-404e-a86d-c83b8e9e5591-kube-api-access-dslqs\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.798237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.798064 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.798237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.798087 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.899289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.899260 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dslqs\" (UniqueName: \"kubernetes.io/projected/4cbbfa87-4335-404e-a86d-c83b8e9e5591-kube-api-access-dslqs\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.899531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.899305 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.899531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.899325 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.900074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.900048 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.901990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.901959 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:10.956206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:10.956178 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dslqs\" (UniqueName: \"kubernetes.io/projected/4cbbfa87-4335-404e-a86d-c83b8e9e5591-kube-api-access-dslqs\") pod \"maas-api-revoke-sub-cf6932e9-j4wk7\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:11.048687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:11.048600 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:12.383774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:12.383737 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:13.063456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:13.063425 5633 scope.go:122] "RemoveContainer" containerID="0b471a8f46d72aab9d0f2f9187557f4e73bbbbe93b69ed1635dd87ddca874181" Nov 01 20:23:13.074861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:13.074836 5633 scope.go:122] "RemoveContainer" containerID="ec4af858e5ceafa3f18a6e4ad654a127e8b66db2432173964780043e5038e81f" Nov 01 20:23:13.085180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:13.085156 5633 scope.go:122] "RemoveContainer" containerID="7c00df074166dcae3f5179a949d3818f7cc1068342c2672af2ef3dd4d4cde606" Nov 01 20:23:13.096096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:13.096073 5633 scope.go:122] "RemoveContainer" containerID="1f95d0100d6a9e94e464857a26ee3670d7847b04ca918f5d78b5f573b6ad12f0" Nov 01 20:23:13.340635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:13.340530 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:14.046097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:14.046063 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf"] Nov 01 20:23:14.095315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:14.095280 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8fd0ce-xh9zf"] Nov 01 20:23:14.467470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:14.467443 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b17db6-5e97-4290-a91e-649391f27ce4" path="/var/lib/kubelet/pods/b6b17db6-5e97-4290-a91e-649391f27ce4/volumes" Nov 01 20:23:15.053102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:15.053062 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:23:16.507903 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:23:16.507850 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:23:16.753849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:16.753792 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:16.754509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:16.754479 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 01 20:23:17.704359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:17.703849 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:17.761349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:17.759539 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 01 20:23:17.761349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:17.760038 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:23:17.806359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:17.795542 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:17.806359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:17.796116 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:23:17.836373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:17.836264 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:18.609358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:18.608938 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:23:18.706373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:18.704430 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:18.706373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:18.705271 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:23:18.784360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:18.771916 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 01 20:23:18.800362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:18.796390 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 01 20:23:19.483363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:19.470914 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:23:19.798362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:19.797064 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:20.358361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:20.349672 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:23:23.141428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:23.141386 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:24.178832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:24.178793 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:24.351628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:24.351591 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:27.783462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:27.783418 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:32.047028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:32.046989 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s"] Nov 01 20:23:32.052463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:32.052433 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cae6732-5mp9s"] Nov 01 20:23:32.469981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:32.469945 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908634c7-1ee8-4dfb-8906-7c7687875aeb" path="/var/lib/kubelet/pods/908634c7-1ee8-4dfb-8906-7c7687875aeb/volumes" Nov 01 20:23:32.490533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:32.490497 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:37.183484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:37.183449 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:40.176511 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:40.176476 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:41.298851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:41.298810 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:43.092867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:43.092834 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:44.496063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:44.496025 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:44.623508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:44.623465 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:46.036771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:46.036738 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:46.162646 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:23:46.162604 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:23:46.514149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:46.514112 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:46.521857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:46.521827 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5"] Nov 01 20:23:46.522686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:46.522649 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" containerID="cri-o://e3e4ad74c860591ffb1152d7b5766662a3e7424a6b46e43166926592d0917da5" gracePeriod=30 Nov 01 20:23:48.026578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:48.026541 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:48.046809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:48.046771 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q"] Nov 01 20:23:48.097277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:48.097225 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a16967dc-vmt9q"] Nov 01 20:23:48.467961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:48.467928 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f89f6a1-1435-45c9-9692-538d897f0f16" path="/var/lib/kubelet/pods/2f89f6a1-1435-45c9-9692-538d897f0f16/volumes" Nov 01 20:23:49.014922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.014794 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:49.085560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.085524 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7"] Nov 01 20:23:49.147127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.147098 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:49.152169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.152138 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc"] Nov 01 20:23:49.488058 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:23:49.488021 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd064b2fd_3070_4712_9f0a_471b3d1e837c.slice/crio-49fa7659078554c79fe6aaeec942ba17c6ea69581c1b44100b95e53868f9d118 WatchSource:0}: Error finding container 49fa7659078554c79fe6aaeec942ba17c6ea69581c1b44100b95e53868f9d118: Status 404 returned error can't find the container with id 49fa7659078554c79fe6aaeec942ba17c6ea69581c1b44100b95e53868f9d118 Nov 01 20:23:49.488241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.488109 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc"] Nov 01 20:23:49.538004 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.537979 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7"] Nov 01 20:23:49.538428 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:23:49.538396 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbbfa87_4335_404e_a86d_c83b8e9e5591.slice/crio-748d1da7ed8ee0c5539768af657657a45a929adf6bc55820cc067b4f26fb34c0 WatchSource:0}: Error finding container 748d1da7ed8ee0c5539768af657657a45a929adf6bc55820cc067b4f26fb34c0: Status 404 returned error can't find the container with id 748d1da7ed8ee0c5539768af657657a45a929adf6bc55820cc067b4f26fb34c0 Nov 01 20:23:49.626793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.626754 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" event={"ID":"d064b2fd-3070-4712-9f0a-471b3d1e837c","Type":"ContainerStarted","Data":"49fa7659078554c79fe6aaeec942ba17c6ea69581c1b44100b95e53868f9d118"} Nov 01 20:23:49.628371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.628305 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" event={"ID":"4cbbfa87-4335-404e-a86d-c83b8e9e5591","Type":"ContainerStarted","Data":"748d1da7ed8ee0c5539768af657657a45a929adf6bc55820cc067b4f26fb34c0"} Nov 01 20:23:49.647038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:49.647002 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:50.638157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:50.638119 5633 generic.go:356] "Generic (PLEG): container finished" podID="4cbbfa87-4335-404e-a86d-c83b8e9e5591" containerID="502ed2e6d15f00c23b821cbf553ce818645fdb072f7086ff205d31bd8c299ffc" exitCode=0 Nov 01 20:23:50.638668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:50.638208 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" event={"ID":"4cbbfa87-4335-404e-a86d-c83b8e9e5591","Type":"ContainerDied","Data":"502ed2e6d15f00c23b821cbf553ce818645fdb072f7086ff205d31bd8c299ffc"} Nov 01 20:23:50.639981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:50.639954 5633 generic.go:356] "Generic (PLEG): container finished" podID="d064b2fd-3070-4712-9f0a-471b3d1e837c" containerID="9812087accf509638554f70b0643903ff055ecd9708b2e1fdc46bac58516f139" exitCode=0 Nov 01 20:23:50.640111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:50.640042 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" event={"ID":"d064b2fd-3070-4712-9f0a-471b3d1e837c","Type":"ContainerDied","Data":"9812087accf509638554f70b0643903ff055ecd9708b2e1fdc46bac58516f139"} Nov 01 20:23:51.651616 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.651585 5633 generic.go:356] "Generic (PLEG): container finished" podID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerID="e3e4ad74c860591ffb1152d7b5766662a3e7424a6b46e43166926592d0917da5" exitCode=0 Nov 01 20:23:51.651987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.651672 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" event={"ID":"31cd5a75-e0c0-479d-aaf0-5a424f40d60c","Type":"ContainerDied","Data":"e3e4ad74c860591ffb1152d7b5766662a3e7424a6b46e43166926592d0917da5"} Nov 01 20:23:51.827855 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.827826 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:23:51.887966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.887938 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:51.891813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.891789 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:51.955525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955496 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-openshift-ca\") pod \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " Nov 01 20:23:51.955732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955572 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-credential-socket\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.955732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955603 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-certs\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.955732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955711 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-openshift-ca\") pod \"d064b2fd-3070-4712-9f0a-471b3d1e837c\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " Nov 01 20:23:51.955899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955752 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-envoy\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.955899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955805 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-xks-ca\") pod \"d064b2fd-3070-4712-9f0a-471b3d1e837c\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " Nov 01 20:23:51.955899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955858 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-xks-ca\") pod \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " Nov 01 20:23:51.955899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955883 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-openshift-ca" pod "4cbbfa87-4335-404e-a86d-c83b8e9e5591" (UID: "4cbbfa87-4335-404e-a86d-c83b8e9e5591"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:51.956104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955909 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-token\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-token\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.956104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.955936 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istiod-ca-cert\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.956104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956013 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-podinfo\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.956104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956048 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbbfa87-4335-404e-a86d-c83b8e9e5591-kube-api-access-dslqs\" (UniqueName: \"kubernetes.io/projected/4cbbfa87-4335-404e-a86d-c83b8e9e5591-kube-api-access-dslqs\") pod \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\" (UID: \"4cbbfa87-4335-404e-a86d-c83b8e9e5591\") " Nov 01 20:23:51.956104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956076 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d064b2fd-3070-4712-9f0a-471b3d1e837c-kube-api-access-nftst\" (UniqueName: \"kubernetes.io/projected/d064b2fd-3070-4712-9f0a-471b3d1e837c-kube-api-access-nftst\") pod \"d064b2fd-3070-4712-9f0a-471b3d1e837c\" (UID: \"d064b2fd-3070-4712-9f0a-471b3d1e837c\") " Nov 01 20:23:51.956378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956120 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-data\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.956378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956195 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-kube-api-access-h2bz6\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-kube-api-access-h2bz6\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.956378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956223 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-socket\") pod \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\" (UID: \"31cd5a75-e0c0-479d-aaf0-5a424f40d60c\") " Nov 01 20:23:51.956378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956255 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-openshift-ca" pod "d064b2fd-3070-4712-9f0a-471b3d1e837c" (UID: "d064b2fd-3070-4712-9f0a-471b3d1e837c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:51.956585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956545 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-credential-socket" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.956736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956719 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-certs" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.956792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956650 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istiod-ca-cert" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:51.956919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956887 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.956919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956910 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.957025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956925 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.957025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956940 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.957025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.956954 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.958614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.958578 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-data" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.959345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.959291 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-socket" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.960491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.960465 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbbfa87-4335-404e-a86d-c83b8e9e5591-kube-api-access-dslqs" pod "4cbbfa87-4335-404e-a86d-c83b8e9e5591" (UID: "4cbbfa87-4335-404e-a86d-c83b8e9e5591"). InnerVolumeSpecName "kube-api-access-dslqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:51.960794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.960752 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-xks-ca" pod "d064b2fd-3070-4712-9f0a-471b3d1e837c" (UID: "d064b2fd-3070-4712-9f0a-471b3d1e837c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:23:51.961096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.961068 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-xks-ca" pod "4cbbfa87-4335-404e-a86d-c83b8e9e5591" (UID: "4cbbfa87-4335-404e-a86d-c83b8e9e5591"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:23:51.961413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.961373 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-token" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:51.961605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.961568 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d064b2fd-3070-4712-9f0a-471b3d1e837c-kube-api-access-nftst" pod "d064b2fd-3070-4712-9f0a-471b3d1e837c" (UID: "d064b2fd-3070-4712-9f0a-471b3d1e837c"). InnerVolumeSpecName "kube-api-access-nftst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:51.961730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.961711 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-kube-api-access-h2bz6" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "kube-api-access-h2bz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:51.961815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.961798 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-envoy" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.961899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:51.961884 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-podinfo" pod "31cd5a75-e0c0-479d-aaf0-5a424f40d60c" (UID: "31cd5a75-e0c0-479d-aaf0-5a424f40d60c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057883 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057914 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2bz6\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-kube-api-access-h2bz6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057926 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057935 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057944 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d064b2fd-3070-4712-9f0a-471b3d1e837c-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057952 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbbfa87-4335-404e-a86d-c83b8e9e5591-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057961 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.057968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057970 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cd5a75-e0c0-479d-aaf0-5a424f40d60c-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.058384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057979 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dslqs\" (UniqueName: \"kubernetes.io/projected/4cbbfa87-4335-404e-a86d-c83b8e9e5591-kube-api-access-dslqs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.058384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.057987 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nftst\" (UniqueName: \"kubernetes.io/projected/d064b2fd-3070-4712-9f0a-471b3d1e837c-kube-api-access-nftst\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.661627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.661592 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" Nov 01 20:23:52.661627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.661620 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7" event={"ID":"4cbbfa87-4335-404e-a86d-c83b8e9e5591","Type":"ContainerDied","Data":"748d1da7ed8ee0c5539768af657657a45a929adf6bc55820cc067b4f26fb34c0"} Nov 01 20:23:52.662206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.661668 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748d1da7ed8ee0c5539768af657657a45a929adf6bc55820cc067b4f26fb34c0" Nov 01 20:23:52.663572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.663541 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" Nov 01 20:23:52.663719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.663589 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc" event={"ID":"d064b2fd-3070-4712-9f0a-471b3d1e837c","Type":"ContainerDied","Data":"49fa7659078554c79fe6aaeec942ba17c6ea69581c1b44100b95e53868f9d118"} Nov 01 20:23:52.663719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.663639 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49fa7659078554c79fe6aaeec942ba17c6ea69581c1b44100b95e53868f9d118" Nov 01 20:23:52.665361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.665306 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" event={"ID":"31cd5a75-e0c0-479d-aaf0-5a424f40d60c","Type":"ContainerDied","Data":"4db7fd4a1a190ec3f9037686cfcec8ba65c19d8fb364af041edc77d5a034cf9a"} Nov 01 20:23:52.665469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.665385 5633 scope.go:122] "RemoveContainer" containerID="e3e4ad74c860591ffb1152d7b5766662a3e7424a6b46e43166926592d0917da5" Nov 01 20:23:52.665469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.665315 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5" Nov 01 20:23:52.885118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.885034 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5"] Nov 01 20:23:52.889062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.889037 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-rwlx5"] Nov 01 20:23:52.994586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:52.994555 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:53.087159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:53.087125 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:23:54.467899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:54.467865 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" path="/var/lib/kubelet/pods/31cd5a75-e0c0-479d-aaf0-5a424f40d60c/volumes" Nov 01 20:23:55.665494 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:55.665460 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:55.840830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:55.840791 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:55.971448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:55.971409 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:56.673280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:56.673229 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:56.872796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:56.872757 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:57.685825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:57.685787 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:58.861572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:58.861539 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:59.016030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:59.015993 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:23:59.047801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:59.047765 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt"] Nov 01 20:23:59.145893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:23:59.145786 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ab19483-mzjbt"] Nov 01 20:24:00.469452 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:00.469409 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d50f956-e566-4dd6-8f67-7856a0365066" path="/var/lib/kubelet/pods/1d50f956-e566-4dd6-8f67-7856a0365066/volumes" Nov 01 20:24:00.924947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:00.924868 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:01.104964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:01.104927 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:01.240621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:01.240577 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:02.310398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.310358 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp"] Nov 01 20:24:02.311165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311142 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d064b2fd-3070-4712-9f0a-471b3d1e837c" containerName="revoke-keys" Nov 01 20:24:02.311275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311168 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d064b2fd-3070-4712-9f0a-471b3d1e837c" containerName="revoke-keys" Nov 01 20:24:02.311275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311192 5633 cpu_manager.go:414] "containerMap: removing container" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" Nov 01 20:24:02.311275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311199 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" Nov 01 20:24:02.311275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311242 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4cbbfa87-4335-404e-a86d-c83b8e9e5591" containerName="revoke-keys" Nov 01 20:24:02.311275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311249 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbbfa87-4335-404e-a86d-c83b8e9e5591" containerName="revoke-keys" Nov 01 20:24:02.311565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311414 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="113e6f55-ced3-4c66-b51d-83f8bc007233" containerName="istio-proxy" Nov 01 20:24:02.311565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311429 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbbfa87-4335-404e-a86d-c83b8e9e5591" containerName="revoke-keys" Nov 01 20:24:02.311565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311449 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d064b2fd-3070-4712-9f0a-471b3d1e837c" containerName="revoke-keys" Nov 01 20:24:02.311565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.311459 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="31cd5a75-e0c0-479d-aaf0-5a424f40d60c" containerName="istio-proxy" Nov 01 20:24:02.315447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.315427 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.370910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.370877 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp"] Nov 01 20:24:02.396927 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.396878 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjz8s\" (UniqueName: \"kubernetes.io/projected/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-kube-api-access-pjz8s\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.397099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.397007 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.397099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.397071 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.414628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.414597 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:24:02.414777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.414632 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:24:02.414777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.414632 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:24:02.416739 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.416725 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:24:02.498500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.498460 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjz8s\" (UniqueName: \"kubernetes.io/projected/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-kube-api-access-pjz8s\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.498712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.498515 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.498712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.498574 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.499296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.499272 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.501136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.501112 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.554890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.554857 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjz8s\" (UniqueName: \"kubernetes.io/projected/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-kube-api-access-pjz8s\") pod \"maas-api-revoke-sub-b05c3623-x66sp\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.629660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.629576 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:02.802243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:02.802205 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:03.011428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.011399 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp"] Nov 01 20:24:03.011530 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:03.011473 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8550344c_79a8_45a3_96f4_a49ad4c2cd4c.slice/crio-60371df91dab6d80b9969c0c9362ead342f232bc15302a581f2a04af3d23c63b WatchSource:0}: Error finding container 60371df91dab6d80b9969c0c9362ead342f232bc15302a581f2a04af3d23c63b: Status 404 returned error can't find the container with id 60371df91dab6d80b9969c0c9362ead342f232bc15302a581f2a04af3d23c63b Nov 01 20:24:03.097095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.097059 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595"] Nov 01 20:24:03.103089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.103062 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.154842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.154812 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-e385637602d5\"" Nov 01 20:24:03.156351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.156313 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595"] Nov 01 20:24:03.211950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.211913 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnm4k\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-kube-api-access-vnm4k\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.212182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.212153 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.212381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.212318 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.212466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.212421 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.313626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.313594 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm4k\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-kube-api-access-vnm4k\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.314087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.313688 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.314087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.313804 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.314087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.313850 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.314302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.314280 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.314455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.314436 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.316465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.316442 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.373387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.373350 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnm4k\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-kube-api-access-vnm4k\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.417912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.417876 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:03.767095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.767051 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" event={"ID":"8550344c-79a8-45a3-96f4-a49ad4c2cd4c","Type":"ContainerStarted","Data":"6a185409f4eb784725d5c9293dd6af3b2b07a69c3aed6de2907d9994e8e48f7e"} Nov 01 20:24:03.767290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.767138 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" event={"ID":"8550344c-79a8-45a3-96f4-a49ad4c2cd4c","Type":"ContainerStarted","Data":"60371df91dab6d80b9969c0c9362ead342f232bc15302a581f2a04af3d23c63b"} Nov 01 20:24:03.800932 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:03.800895 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb454fc0_1487_43c5_b037_58dc8bbd4071.slice/crio-61411992de70b12a8f9eaaed809b4e47985fe3e11a2bc87047d4e5f324cc6265 WatchSource:0}: Error finding container 61411992de70b12a8f9eaaed809b4e47985fe3e11a2bc87047d4e5f324cc6265: Status 404 returned error can't find the container with id 61411992de70b12a8f9eaaed809b4e47985fe3e11a2bc87047d4e5f324cc6265 Nov 01 20:24:03.801430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:03.801404 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595"] Nov 01 20:24:04.043869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.043758 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" podStartSLOduration=2.043740845 podStartE2EDuration="2.043740845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:03.878899246 +0000 UTC m=+1251.553158500" watchObservedRunningTime="2026-11-01 20:24:04.043740845 +0000 UTC m=+1251.718000100" Nov 01 20:24:04.045946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.045914 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z"] Nov 01 20:24:04.050735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.050709 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f044b9-gp64z"] Nov 01 20:24:04.467591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.467558 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aeae86d-b416-4c82-ab20-ed96588c1c68" path="/var/lib/kubelet/pods/2aeae86d-b416-4c82-ab20-ed96588c1c68/volumes" Nov 01 20:24:04.777362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.777253 5633 generic.go:356] "Generic (PLEG): container finished" podID="8550344c-79a8-45a3-96f4-a49ad4c2cd4c" containerID="6a185409f4eb784725d5c9293dd6af3b2b07a69c3aed6de2907d9994e8e48f7e" exitCode=0 Nov 01 20:24:04.777536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.777369 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" event={"ID":"8550344c-79a8-45a3-96f4-a49ad4c2cd4c","Type":"ContainerDied","Data":"6a185409f4eb784725d5c9293dd6af3b2b07a69c3aed6de2907d9994e8e48f7e"} Nov 01 20:24:04.779154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.779124 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" event={"ID":"db454fc0-1487-43c5-b037-58dc8bbd4071","Type":"ContainerStarted","Data":"02feecc0e56d0c4fbb7f29bbb40d9f6e81d40adf4c1d024b6872ebf433cdff7c"} Nov 01 20:24:04.779282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.779174 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" event={"ID":"db454fc0-1487-43c5-b037-58dc8bbd4071","Type":"ContainerStarted","Data":"61411992de70b12a8f9eaaed809b4e47985fe3e11a2bc87047d4e5f324cc6265"} Nov 01 20:24:04.779282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:04.779221 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:05.019785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.019729 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" podStartSLOduration=2.019714326 podStartE2EDuration="2.019714326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:05.015797259 +0000 UTC m=+1252.690056522" watchObservedRunningTime="2026-11-01 20:24:05.019714326 +0000 UTC m=+1252.693973580" Nov 01 20:24:05.505594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.505562 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq"] Nov 01 20:24:05.509977 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.509944 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.564989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.564952 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq"] Nov 01 20:24:05.643071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.643031 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpkg8\" (UniqueName: \"kubernetes.io/projected/c97dd085-e0fb-476c-b164-4e30174c1352-kube-api-access-dpkg8\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.643290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.643139 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.643290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.643221 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.744952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.744916 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpkg8\" (UniqueName: \"kubernetes.io/projected/c97dd085-e0fb-476c-b164-4e30174c1352-kube-api-access-dpkg8\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.745128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.745002 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.745128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.745089 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.745779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.745760 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.747513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.747493 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.802044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.801962 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpkg8\" (UniqueName: \"kubernetes.io/projected/c97dd085-e0fb-476c-b164-4e30174c1352-kube-api-access-dpkg8\") pod \"maas-api-revoke-sub-47c84570-z4kgq\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.824162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.824129 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:05.825455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:05.825395 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:06.016633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.011026 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:06.029657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.029628 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:06.164939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.164829 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-kube-api-access-pjz8s\" (UniqueName: \"kubernetes.io/projected/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-kube-api-access-pjz8s\") pod \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " Nov 01 20:24:06.165567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.165454 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-xks-ca\") pod \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " Nov 01 20:24:06.165567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.165520 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-openshift-ca\") pod \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\" (UID: \"8550344c-79a8-45a3-96f4-a49ad4c2cd4c\") " Nov 01 20:24:06.169302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.169262 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-openshift-ca" pod "8550344c-79a8-45a3-96f4-a49ad4c2cd4c" (UID: "8550344c-79a8-45a3-96f4-a49ad4c2cd4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:06.169945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.169918 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:06.200724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.200670 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-xks-ca" pod "8550344c-79a8-45a3-96f4-a49ad4c2cd4c" (UID: "8550344c-79a8-45a3-96f4-a49ad4c2cd4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:06.206217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.206139 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-kube-api-access-pjz8s" pod "8550344c-79a8-45a3-96f4-a49ad4c2cd4c" (UID: "8550344c-79a8-45a3-96f4-a49ad4c2cd4c"). InnerVolumeSpecName "kube-api-access-pjz8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:06.242846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.242797 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7"] Nov 01 20:24:06.243562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.243541 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8550344c-79a8-45a3-96f4-a49ad4c2cd4c" containerName="revoke-keys" Nov 01 20:24:06.243675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.243564 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8550344c-79a8-45a3-96f4-a49ad4c2cd4c" containerName="revoke-keys" Nov 01 20:24:06.243742 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.243728 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8550344c-79a8-45a3-96f4-a49ad4c2cd4c" containerName="revoke-keys" Nov 01 20:24:06.247714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.247689 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.260972 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:06.260933 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97dd085_e0fb_476c_b164_4e30174c1352.slice/crio-7c431f65ae801182b0e7d8b7f501c3bdce5323d53759cf2cf22b736875623b8e WatchSource:0}: Error finding container 7c431f65ae801182b0e7d8b7f501c3bdce5323d53759cf2cf22b736875623b8e: Status 404 returned error can't find the container with id 7c431f65ae801182b0e7d8b7f501c3bdce5323d53759cf2cf22b736875623b8e Nov 01 20:24:06.261790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.261762 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq"] Nov 01 20:24:06.271047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.271022 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:06.271047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.271048 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjz8s\" (UniqueName: \"kubernetes.io/projected/8550344c-79a8-45a3-96f4-a49ad4c2cd4c-kube-api-access-pjz8s\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:06.299803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.299769 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2a4e4430-dockercfg-qpf72\"" Nov 01 20:24:06.299990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.299851 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2a4e4430\"" Nov 01 20:24:06.302914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.302881 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7"] Nov 01 20:24:06.373359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.372431 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3a0214-4207-4963-a48b-c070128bd064-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.373359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.372541 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jkhp\" (UniqueName: \"kubernetes.io/projected/9b3a0214-4207-4963-a48b-c070128bd064-kube-api-access-7jkhp\") pod \"payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.475212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.475177 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3a0214-4207-4963-a48b-c070128bd064-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.475425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.475277 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jkhp\" (UniqueName: \"kubernetes.io/projected/9b3a0214-4207-4963-a48b-c070128bd064-kube-api-access-7jkhp\") pod \"payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.476180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.476147 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3a0214-4207-4963-a48b-c070128bd064-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.532127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.532091 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jkhp\" (UniqueName: \"kubernetes.io/projected/9b3a0214-4207-4963-a48b-c070128bd064-kube-api-access-7jkhp\") pod \"payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.563580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.563541 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:06.801831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.801792 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" event={"ID":"c97dd085-e0fb-476c-b164-4e30174c1352","Type":"ContainerStarted","Data":"ecf26408aab4ebad0a3c6057f57cba175731f47ebcb9c43bb81f5522a9ba26d1"} Nov 01 20:24:06.802036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.801852 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" event={"ID":"c97dd085-e0fb-476c-b164-4e30174c1352","Type":"ContainerStarted","Data":"7c431f65ae801182b0e7d8b7f501c3bdce5323d53759cf2cf22b736875623b8e"} Nov 01 20:24:06.803539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.803515 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" event={"ID":"8550344c-79a8-45a3-96f4-a49ad4c2cd4c","Type":"ContainerDied","Data":"60371df91dab6d80b9969c0c9362ead342f232bc15302a581f2a04af3d23c63b"} Nov 01 20:24:06.803658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.803556 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60371df91dab6d80b9969c0c9362ead342f232bc15302a581f2a04af3d23c63b" Nov 01 20:24:06.803658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.803524 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp" Nov 01 20:24:06.920729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.920537 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" podStartSLOduration=1.92052232 podStartE2EDuration="1.92052232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:06.915325443 +0000 UTC m=+1254.589584698" watchObservedRunningTime="2026-11-01 20:24:06.92052232 +0000 UTC m=+1254.594781588" Nov 01 20:24:06.943820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:06.943790 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7"] Nov 01 20:24:06.944213 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:06.944188 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3a0214_4207_4963_a48b_c070128bd064.slice/crio-4fdb912f250868f61dbeff1cb3d430d2b0b5a243bdc3a6b14493cdb041ed6231 WatchSource:0}: Error finding container 4fdb912f250868f61dbeff1cb3d430d2b0b5a243bdc3a6b14493cdb041ed6231: Status 404 returned error can't find the container with id 4fdb912f250868f61dbeff1cb3d430d2b0b5a243bdc3a6b14493cdb041ed6231 Nov 01 20:24:07.683641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:07.683546 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:07.826003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:07.825969 5633 generic.go:356] "Generic (PLEG): container finished" podID="c97dd085-e0fb-476c-b164-4e30174c1352" containerID="ecf26408aab4ebad0a3c6057f57cba175731f47ebcb9c43bb81f5522a9ba26d1" exitCode=0 Nov 01 20:24:07.826173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:07.826054 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" event={"ID":"c97dd085-e0fb-476c-b164-4e30174c1352","Type":"ContainerDied","Data":"ecf26408aab4ebad0a3c6057f57cba175731f47ebcb9c43bb81f5522a9ba26d1"} Nov 01 20:24:07.827999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:07.827969 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" event={"ID":"9b3a0214-4207-4963-a48b-c070128bd064","Type":"ContainerStarted","Data":"14a46b60b1802bf190df52d5d43761677667d18969efe8e74beec853a98ff15c"} Nov 01 20:24:07.828154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:07.828019 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" event={"ID":"9b3a0214-4207-4963-a48b-c070128bd064","Type":"ContainerStarted","Data":"4fdb912f250868f61dbeff1cb3d430d2b0b5a243bdc3a6b14493cdb041ed6231"} Nov 01 20:24:07.828154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:07.828062 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:08.051312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:08.051257 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" podStartSLOduration=2.051242528 podStartE2EDuration="2.051242528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:08.048190602 +0000 UTC m=+1255.722449860" watchObservedRunningTime="2026-11-01 20:24:08.051242528 +0000 UTC m=+1255.725501783" Nov 01 20:24:09.012167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.012140 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:09.106851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.106758 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97dd085-e0fb-476c-b164-4e30174c1352-kube-api-access-dpkg8\" (UniqueName: \"kubernetes.io/projected/c97dd085-e0fb-476c-b164-4e30174c1352-kube-api-access-dpkg8\") pod \"c97dd085-e0fb-476c-b164-4e30174c1352\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " Nov 01 20:24:09.106851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.106816 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-xks-ca\") pod \"c97dd085-e0fb-476c-b164-4e30174c1352\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " Nov 01 20:24:09.107102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.106879 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-openshift-ca\") pod \"c97dd085-e0fb-476c-b164-4e30174c1352\" (UID: \"c97dd085-e0fb-476c-b164-4e30174c1352\") " Nov 01 20:24:09.107299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.107273 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-openshift-ca" pod "c97dd085-e0fb-476c-b164-4e30174c1352" (UID: "c97dd085-e0fb-476c-b164-4e30174c1352"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:09.109097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.109072 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97dd085-e0fb-476c-b164-4e30174c1352-kube-api-access-dpkg8" pod "c97dd085-e0fb-476c-b164-4e30174c1352" (UID: "c97dd085-e0fb-476c-b164-4e30174c1352"). InnerVolumeSpecName "kube-api-access-dpkg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:09.109184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.109108 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-xks-ca" pod "c97dd085-e0fb-476c-b164-4e30174c1352" (UID: "c97dd085-e0fb-476c-b164-4e30174c1352"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:09.208069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.208027 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:09.208069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.208061 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97dd085-e0fb-476c-b164-4e30174c1352-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:09.208069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.208075 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpkg8\" (UniqueName: \"kubernetes.io/projected/c97dd085-e0fb-476c-b164-4e30174c1352-kube-api-access-dpkg8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:09.846344 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.846312 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" Nov 01 20:24:09.846554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.846310 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq" event={"ID":"c97dd085-e0fb-476c-b164-4e30174c1352","Type":"ContainerDied","Data":"7c431f65ae801182b0e7d8b7f501c3bdce5323d53759cf2cf22b736875623b8e"} Nov 01 20:24:09.846554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:09.846452 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c431f65ae801182b0e7d8b7f501c3bdce5323d53759cf2cf22b736875623b8e" Nov 01 20:24:12.516473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:12.516429 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:13.083460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.083409 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg"] Nov 01 20:24:13.104704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.104668 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c97dd085-e0fb-476c-b164-4e30174c1352" containerName="revoke-keys" Nov 01 20:24:13.104951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.104930 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97dd085-e0fb-476c-b164-4e30174c1352" containerName="revoke-keys" Nov 01 20:24:13.105256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.105237 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97dd085-e0fb-476c-b164-4e30174c1352" containerName="revoke-keys" Nov 01 20:24:13.125831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.125802 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.190357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.162946 5633 scope.go:122] "RemoveContainer" containerID="73106f3246b74a8dbcc5479f7becef239292624f72e199d45332508be41073db" Nov 01 20:24:13.224725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.193545 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:24:13.224725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.193670 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:24:13.224725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.193781 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:24:13.243389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.226561 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.243389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.226636 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.243389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.226795 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlkb\" (UniqueName: \"kubernetes.io/projected/f3633cce-238d-4572-a3fe-5ce6ee71db3f-kube-api-access-jzlkb\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.258388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.243932 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:24:13.442446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.431009 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg"] Nov 01 20:24:13.491782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.491747 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.492121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.492097 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.493291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.493264 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlkb\" (UniqueName: \"kubernetes.io/projected/f3633cce-238d-4572-a3fe-5ce6ee71db3f-kube-api-access-jzlkb\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.493925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.493032 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.588608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.585494 5633 scope.go:122] "RemoveContainer" containerID="906720b9e7610aa4bfebf6da1d70f1dc22841a99a312d4faa646a5b92db0b539" Nov 01 20:24:13.611365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.602523 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlkb\" (UniqueName: \"kubernetes.io/projected/f3633cce-238d-4572-a3fe-5ce6ee71db3f-kube-api-access-jzlkb\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.633723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.633639 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ccd6b38-6j6xg\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.684154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.684124 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:13.879565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:13.878379 5633 scope.go:122] "RemoveContainer" containerID="e5e52a3bcf8d87d6ef514a530d4425736f0c8905010450b6ed254a20bc57dc35" Nov 01 20:24:14.112359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:14.109183 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg"] Nov 01 20:24:14.123359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:14.120057 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048ffa5e-mlqbg"] Nov 01 20:24:14.130265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:14.129900 5633 scope.go:122] "RemoveContainer" containerID="4ae9abf5d64219e670c2355eaa8d8d6111649c917a31cbc0f373ece6b819f8ff" Nov 01 20:24:14.470056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:14.470021 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14856dd8-8d3a-492d-8752-d73dec42d4e4" path="/var/lib/kubelet/pods/14856dd8-8d3a-492d-8752-d73dec42d4e4/volumes" Nov 01 20:24:14.545701 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:14.545663 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3633cce_238d_4572_a3fe_5ce6ee71db3f.slice/crio-2969394e80c616d3fcf384187a14b1bb392d70cec5a0d126f69730c80cc1dc22 WatchSource:0}: Error finding container 2969394e80c616d3fcf384187a14b1bb392d70cec5a0d126f69730c80cc1dc22: Status 404 returned error can't find the container with id 2969394e80c616d3fcf384187a14b1bb392d70cec5a0d126f69730c80cc1dc22 Nov 01 20:24:14.546067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:14.546043 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg"] Nov 01 20:24:15.134622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:15.134521 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" event={"ID":"f3633cce-238d-4572-a3fe-5ce6ee71db3f","Type":"ContainerStarted","Data":"efa2f0a960c8ce0257c11107a101be3fd7f6d4503d4e41cf9edeb1f8974c622d"} Nov 01 20:24:15.134622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:15.134576 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" event={"ID":"f3633cce-238d-4572-a3fe-5ce6ee71db3f","Type":"ContainerStarted","Data":"2969394e80c616d3fcf384187a14b1bb392d70cec5a0d126f69730c80cc1dc22"} Nov 01 20:24:15.249686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:15.249628 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" podStartSLOduration=2.249612965 podStartE2EDuration="2.249612965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:15.247009266 +0000 UTC m=+1262.921268522" watchObservedRunningTime="2026-11-01 20:24:15.249612965 +0000 UTC m=+1262.923872255" Nov 01 20:24:15.645254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:15.645212 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:15.796179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:15.796144 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:15.857359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:15.850315 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:15.965824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:15.965788 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw"] Nov 01 20:24:16.151659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:16.151566 5633 generic.go:356] "Generic (PLEG): container finished" podID="f3633cce-238d-4572-a3fe-5ce6ee71db3f" containerID="efa2f0a960c8ce0257c11107a101be3fd7f6d4503d4e41cf9edeb1f8974c622d" exitCode=0 Nov 01 20:24:16.151659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:16.151643 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" event={"ID":"f3633cce-238d-4572-a3fe-5ce6ee71db3f","Type":"ContainerDied","Data":"efa2f0a960c8ce0257c11107a101be3fd7f6d4503d4e41cf9edeb1f8974c622d"} Nov 01 20:24:16.168509 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:24:16.168470 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:24:17.340906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.340876 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:17.442802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.442714 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-openshift-ca\") pod \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " Nov 01 20:24:17.442802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.442755 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-xks-ca\") pod \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " Nov 01 20:24:17.443031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.442809 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3633cce-238d-4572-a3fe-5ce6ee71db3f-kube-api-access-jzlkb\" (UniqueName: \"kubernetes.io/projected/f3633cce-238d-4572-a3fe-5ce6ee71db3f-kube-api-access-jzlkb\") pod \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\" (UID: \"f3633cce-238d-4572-a3fe-5ce6ee71db3f\") " Nov 01 20:24:17.443147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.443125 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-openshift-ca" pod "f3633cce-238d-4572-a3fe-5ce6ee71db3f" (UID: "f3633cce-238d-4572-a3fe-5ce6ee71db3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:17.443455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.443431 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:17.445182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.445153 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-xks-ca" pod "f3633cce-238d-4572-a3fe-5ce6ee71db3f" (UID: "f3633cce-238d-4572-a3fe-5ce6ee71db3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:17.445306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.445288 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3633cce-238d-4572-a3fe-5ce6ee71db3f-kube-api-access-jzlkb" pod "f3633cce-238d-4572-a3fe-5ce6ee71db3f" (UID: "f3633cce-238d-4572-a3fe-5ce6ee71db3f"). InnerVolumeSpecName "kube-api-access-jzlkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:17.544530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.544490 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3633cce-238d-4572-a3fe-5ce6ee71db3f-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:17.544530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.544520 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzlkb\" (UniqueName: \"kubernetes.io/projected/f3633cce-238d-4572-a3fe-5ce6ee71db3f-kube-api-access-jzlkb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:17.586967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:17.586921 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:18.175207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.175177 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" Nov 01 20:24:18.175404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.175209 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg" event={"ID":"f3633cce-238d-4572-a3fe-5ce6ee71db3f","Type":"ContainerDied","Data":"2969394e80c616d3fcf384187a14b1bb392d70cec5a0d126f69730c80cc1dc22"} Nov 01 20:24:18.175404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.175266 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2969394e80c616d3fcf384187a14b1bb392d70cec5a0d126f69730c80cc1dc22" Nov 01 20:24:18.345130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.345095 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:18.839607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.839573 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:24:18.869890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.869855 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999"] Nov 01 20:24:18.870563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.870548 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f3633cce-238d-4572-a3fe-5ce6ee71db3f" containerName="revoke-keys" Nov 01 20:24:18.870625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.870565 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3633cce-238d-4572-a3fe-5ce6ee71db3f" containerName="revoke-keys" Nov 01 20:24:18.870696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.870686 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3633cce-238d-4572-a3fe-5ce6ee71db3f" containerName="revoke-keys" Nov 01 20:24:18.873938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.873924 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:18.928572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.928540 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 20:24:18.932552 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.932519 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999"] Nov 01 20:24:18.940738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.940698 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 20:24:18.940738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.940702 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 20:24:18.940951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.940762 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-fbdbcfd54e6f\"" Nov 01 20:24:18.940951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.940792 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 20:24:18.940951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.940819 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-dqptr\"" Nov 01 20:24:18.968167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.968129 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:18.968363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.968244 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:18.968363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.968322 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:18.968561 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:18.968538 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddks\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-kube-api-access-9ddks\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.070276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.070240 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddks\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-kube-api-access-9ddks\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.070485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.070368 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.070485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.070408 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.070485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.070437 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.071153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.071126 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.071268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.071210 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.073048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.073024 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.126668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.126596 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddks\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-kube-api-access-9ddks\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-f456pv999\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.188419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.188324 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:19.612291 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:19.612253 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec94b85a_3d99_42b0_b540_be2d8514c773.slice/crio-74da876c33914b7543ac7250085edb8bf50ff5d0a1594f41826b1ca5d45cf5f7 WatchSource:0}: Error finding container 74da876c33914b7543ac7250085edb8bf50ff5d0a1594f41826b1ca5d45cf5f7: Status 404 returned error can't find the container with id 74da876c33914b7543ac7250085edb8bf50ff5d0a1594f41826b1ca5d45cf5f7 Nov 01 20:24:19.612945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:19.612923 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999"] Nov 01 20:24:20.192839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.192799 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" event={"ID":"ec94b85a-3d99-42b0-b540-be2d8514c773","Type":"ContainerStarted","Data":"461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d"} Nov 01 20:24:20.193056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.192862 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" event={"ID":"ec94b85a-3d99-42b0-b540-be2d8514c773","Type":"ContainerStarted","Data":"74da876c33914b7543ac7250085edb8bf50ff5d0a1594f41826b1ca5d45cf5f7"} Nov 01 20:24:20.193056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.192892 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:20.309805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.309750 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" podStartSLOduration=2.309736472 podStartE2EDuration="2.309736472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:20.305216949 +0000 UTC m=+1267.979476210" watchObservedRunningTime="2026-11-01 20:24:20.309736472 +0000 UTC m=+1267.983995726" Nov 01 20:24:20.712945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.712910 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt"] Nov 01 20:24:20.717495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.717468 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.770385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.769990 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-cd18c1528ab8\"" Nov 01 20:24:20.775815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.775782 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt"] Nov 01 20:24:20.806851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.806818 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.807050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.806868 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.807050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.806983 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.807050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.807030 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9x9\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-kube-api-access-6v9x9\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.908510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.908470 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.908694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.908539 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.908694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.908581 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.908794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.908716 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9x9\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-kube-api-access-6v9x9\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.909201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.909180 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.909307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.909290 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.911212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.911192 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.965496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.965425 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9x9\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-kube-api-access-6v9x9\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:20.967114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:20.967080 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" podUID="fe142b44-da56-4c21-b797-944c98c0362a" containerName="payload-processing" containerID="cri-o://4962d4de0e7eabea7a370e985ab76aa066ab8003b763dd6f9e26ac3a14d3aa0a" gracePeriod=25 Nov 01 20:24:21.028933 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:21.028865 5633 logging.go:55] [core] [Channel #1028 SubChannel #1029]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.124:50052", ServerName: "10.134.0.124:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.124:50052: connect: connection refused" Nov 01 20:24:21.032995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.032969 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:21.205415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.205364 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" event={"ID":"fe142b44-da56-4c21-b797-944c98c0362a","Type":"ContainerDied","Data":"4962d4de0e7eabea7a370e985ab76aa066ab8003b763dd6f9e26ac3a14d3aa0a"} Nov 01 20:24:21.205579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.205466 5633 generic.go:356] "Generic (PLEG): container finished" podID="fe142b44-da56-4c21-b797-944c98c0362a" containerID="4962d4de0e7eabea7a370e985ab76aa066ab8003b763dd6f9e26ac3a14d3aa0a" exitCode=0 Nov 01 20:24:21.277084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.277054 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:24:21.315311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315275 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-routing-overlay-volume\") pod \"fe142b44-da56-4c21-b797-944c98c0362a\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " Nov 01 20:24:21.315513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315321 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-plugins-config-volume\") pod \"fe142b44-da56-4c21-b797-944c98c0362a\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " Nov 01 20:24:21.315513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315420 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-kube-api-access-jm9vw\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-kube-api-access-jm9vw\") pod \"fe142b44-da56-4c21-b797-944c98c0362a\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " Nov 01 20:24:21.315513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315491 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-provider-credentials\") pod \"fe142b44-da56-4c21-b797-944c98c0362a\" (UID: \"fe142b44-da56-4c21-b797-944c98c0362a\") " Nov 01 20:24:21.315689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315660 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-routing-overlay-volume" pod "fe142b44-da56-4c21-b797-944c98c0362a" (UID: "fe142b44-da56-4c21-b797-944c98c0362a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:21.315777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315759 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-plugins-config-volume" pod "fe142b44-da56-4c21-b797-944c98c0362a" (UID: "fe142b44-da56-4c21-b797-944c98c0362a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:21.315979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315966 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:21.316014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.315981 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe142b44-da56-4c21-b797-944c98c0362a-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:21.317897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.317875 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-kube-api-access-jm9vw" pod "fe142b44-da56-4c21-b797-944c98c0362a" (UID: "fe142b44-da56-4c21-b797-944c98c0362a"). InnerVolumeSpecName "kube-api-access-jm9vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:21.317897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.317884 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-provider-credentials" pod "fe142b44-da56-4c21-b797-944c98c0362a" (UID: "fe142b44-da56-4c21-b797-944c98c0362a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:21.417296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.417261 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt"] Nov 01 20:24:21.417472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.417364 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm9vw\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-kube-api-access-jm9vw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:21.417472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:21.417395 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe142b44-da56-4c21-b797-944c98c0362a-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:21.418523 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:21.418493 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd39436_7442_42f3_aa2d_5abb664f2415.slice/crio-451556235f0ad963ca98b31d5395e2785294b3132952f1c54772de9c4ad9b0af WatchSource:0}: Error finding container 451556235f0ad963ca98b31d5395e2785294b3132952f1c54772de9c4ad9b0af: Status 404 returned error can't find the container with id 451556235f0ad963ca98b31d5395e2785294b3132952f1c54772de9c4ad9b0af Nov 01 20:24:22.028618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.028576 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" podUID="fe142b44-da56-4c21-b797-944c98c0362a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.124:50052\" within 1s: context deadline exceeded" Nov 01 20:24:22.215493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.215458 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" Nov 01 20:24:22.215724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.215493 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw" event={"ID":"fe142b44-da56-4c21-b797-944c98c0362a","Type":"ContainerDied","Data":"b546ad3acf4f96df33cfcf58f9649a876272a4c1c8ddf4727cb33e2f8d2e41b6"} Nov 01 20:24:22.215724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.215576 5633 scope.go:122] "RemoveContainer" containerID="4962d4de0e7eabea7a370e985ab76aa066ab8003b763dd6f9e26ac3a14d3aa0a" Nov 01 20:24:22.217798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.217773 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" event={"ID":"4dd39436-7442-42f3-aa2d-5abb664f2415","Type":"ContainerStarted","Data":"c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f"} Nov 01 20:24:22.217924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.217817 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" event={"ID":"4dd39436-7442-42f3-aa2d-5abb664f2415","Type":"ContainerStarted","Data":"451556235f0ad963ca98b31d5395e2785294b3132952f1c54772de9c4ad9b0af"} Nov 01 20:24:22.217924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.217846 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:22.291488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.291391 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:22.383635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.383599 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw"] Nov 01 20:24:22.388292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.388261 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d48zwtw"] Nov 01 20:24:22.468255 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.468223 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe142b44-da56-4c21-b797-944c98c0362a" path="/var/lib/kubelet/pods/fe142b44-da56-4c21-b797-944c98c0362a/volumes" Nov 01 20:24:22.498808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:22.498751 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" podStartSLOduration=2.498736179 podStartE2EDuration="2.498736179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:22.495179162 +0000 UTC m=+1270.169438398" watchObservedRunningTime="2026-11-01 20:24:22.498736179 +0000 UTC m=+1270.172995433" Nov 01 20:24:23.049079 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:23.049043 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v"] Nov 01 20:24:23.053030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:23.053009 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bde0792-sjp8v"] Nov 01 20:24:24.471021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:24.470983 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42cc7b1-6701-43d1-a9ce-81e634d32e43" path="/var/lib/kubelet/pods/f42cc7b1-6701-43d1-a9ce-81e634d32e43/volumes" Nov 01 20:24:24.517000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:24.516952 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:26.189357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:26.189180 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:24:27.288367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.288269 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:27.370360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.370260 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c"] Nov 01 20:24:27.376359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.372981 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fe142b44-da56-4c21-b797-944c98c0362a" containerName="payload-processing" Nov 01 20:24:27.376359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.373011 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe142b44-da56-4c21-b797-944c98c0362a" containerName="payload-processing" Nov 01 20:24:27.376359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.373162 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe142b44-da56-4c21-b797-944c98c0362a" containerName="payload-processing" Nov 01 20:24:27.414360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.414251 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:27.460485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.429736 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c"] Nov 01 20:24:27.626510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.560714 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:24:27.626510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.560857 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:24:27.626510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.570383 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:24:27.626510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.570764 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:24:27.844372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.822107 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:27.844372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.822387 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:27.844372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.822544 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55tqw\" (UniqueName: \"kubernetes.io/projected/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-kube-api-access-55tqw\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:27.939184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.915390 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:27.974460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.959359 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999"] Nov 01 20:24:27.994067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.993065 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:27.994067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.993200 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55tqw\" (UniqueName: \"kubernetes.io/projected/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-kube-api-access-55tqw\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:27.994067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:27.993246 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:28.064360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:28.014280 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:28.094595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:28.074536 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:28.120355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:28.100563 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55tqw\" (UniqueName: \"kubernetes.io/projected/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-kube-api-access-55tqw\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:28.123895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:28.123820 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ebe6afc-lvx5c\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:28.353365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:28.350311 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:29.640507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:29.640466 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c"] Nov 01 20:24:29.679636 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:29.679573 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea602a2d_1c12_4f45_b0e9_ae7809a5820b.slice/crio-dec6217eb1d958d3283c1afdcbe40bf27f2d53e041ed0eacd613d054a66e56d9 WatchSource:0}: Error finding container dec6217eb1d958d3283c1afdcbe40bf27f2d53e041ed0eacd613d054a66e56d9: Status 404 returned error can't find the container with id dec6217eb1d958d3283c1afdcbe40bf27f2d53e041ed0eacd613d054a66e56d9 Nov 01 20:24:30.589897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:30.589808 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" event={"ID":"ea602a2d-1c12-4f45-b0e9-ae7809a5820b","Type":"ContainerStarted","Data":"ce09a11f5223c1193c543fd64c849607c73ce47331cec66fbb036898a848de5e"} Nov 01 20:24:30.589897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:30.589880 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" event={"ID":"ea602a2d-1c12-4f45-b0e9-ae7809a5820b","Type":"ContainerStarted","Data":"dec6217eb1d958d3283c1afdcbe40bf27f2d53e041ed0eacd613d054a66e56d9"} Nov 01 20:24:30.706116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:30.704486 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" podStartSLOduration=3.704466197 podStartE2EDuration="3.704466197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:30.70360226 +0000 UTC m=+1278.377861516" watchObservedRunningTime="2026-11-01 20:24:30.704466197 +0000 UTC m=+1278.378725453" Nov 01 20:24:31.598815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:31.598780 5633 generic.go:356] "Generic (PLEG): container finished" podID="ea602a2d-1c12-4f45-b0e9-ae7809a5820b" containerID="ce09a11f5223c1193c543fd64c849607c73ce47331cec66fbb036898a848de5e" exitCode=0 Nov 01 20:24:31.599021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:31.598869 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" event={"ID":"ea602a2d-1c12-4f45-b0e9-ae7809a5820b","Type":"ContainerDied","Data":"ce09a11f5223c1193c543fd64c849607c73ce47331cec66fbb036898a848de5e"} Nov 01 20:24:31.968894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:31.968860 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq"] Nov 01 20:24:31.973471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:31.973446 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.029677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.029631 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq"] Nov 01 20:24:32.046965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.046931 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv"] Nov 01 20:24:32.051088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.051060 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57294611-x26pv"] Nov 01 20:24:32.108659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.108624 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.108836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.108666 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.108836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.108790 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xd48\" (UniqueName: \"kubernetes.io/projected/f25fb041-260d-4ec0-86fe-79cef548655f-kube-api-access-5xd48\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.209499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.209461 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.209668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.209579 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xd48\" (UniqueName: \"kubernetes.io/projected/f25fb041-260d-4ec0-86fe-79cef548655f-kube-api-access-5xd48\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.209785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.209770 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.210415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.210390 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.212103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.212081 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.265984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.265906 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xd48\" (UniqueName: \"kubernetes.io/projected/f25fb041-260d-4ec0-86fe-79cef548655f-kube-api-access-5xd48\") pod \"maas-api-revoke-sub-bdb76be5-fbrsq\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.288987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.288954 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:32.469085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.469056 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3252f4e-a48c-4fa4-a55b-1e9ba4580db6" path="/var/lib/kubelet/pods/d3252f4e-a48c-4fa4-a55b-1e9ba4580db6/volumes" Nov 01 20:24:32.679214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.679174 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq"] Nov 01 20:24:32.679715 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:32.679677 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25fb041_260d_4ec0_86fe_79cef548655f.slice/crio-bff506a9ff90154058c53de9c0be3e80bc83d7ee05d8d584e612bf7c054a52a1 WatchSource:0}: Error finding container bff506a9ff90154058c53de9c0be3e80bc83d7ee05d8d584e612bf7c054a52a1: Status 404 returned error can't find the container with id bff506a9ff90154058c53de9c0be3e80bc83d7ee05d8d584e612bf7c054a52a1 Nov 01 20:24:32.691044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.690996 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:32.764396 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.764362 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:32.817268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.817229 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-kube-api-access-55tqw\" (UniqueName: \"kubernetes.io/projected/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-kube-api-access-55tqw\") pod \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " Nov 01 20:24:32.817506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.817411 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-openshift-ca\") pod \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " Nov 01 20:24:32.817506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.817472 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-xks-ca\") pod \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\" (UID: \"ea602a2d-1c12-4f45-b0e9-ae7809a5820b\") " Nov 01 20:24:32.817774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.817750 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-openshift-ca" pod "ea602a2d-1c12-4f45-b0e9-ae7809a5820b" (UID: "ea602a2d-1c12-4f45-b0e9-ae7809a5820b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:32.817938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.817924 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:32.819678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.819653 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-xks-ca" pod "ea602a2d-1c12-4f45-b0e9-ae7809a5820b" (UID: "ea602a2d-1c12-4f45-b0e9-ae7809a5820b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:32.819784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.819735 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-kube-api-access-55tqw" pod "ea602a2d-1c12-4f45-b0e9-ae7809a5820b" (UID: "ea602a2d-1c12-4f45-b0e9-ae7809a5820b"). InnerVolumeSpecName "kube-api-access-55tqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:32.918975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.918941 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:32.918975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.918977 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55tqw\" (UniqueName: \"kubernetes.io/projected/ea602a2d-1c12-4f45-b0e9-ae7809a5820b-kube-api-access-55tqw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:32.960226 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:32.960184 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" podUID="ec94b85a-3d99-42b0-b540-be2d8514c773" containerName="payload-processing" containerID="cri-o://461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d" gracePeriod=25 Nov 01 20:24:33.231920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.231895 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:24:33.251821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.251790 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:33.323918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.323884 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-plugins-config-volume\") pod \"ec94b85a-3d99-42b0-b540-be2d8514c773\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " Nov 01 20:24:33.324105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.324021 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-routing-overlay-volume\") pod \"ec94b85a-3d99-42b0-b540-be2d8514c773\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " Nov 01 20:24:33.324105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.324080 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-provider-credentials\") pod \"ec94b85a-3d99-42b0-b540-be2d8514c773\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " Nov 01 20:24:33.324208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.324105 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-kube-api-access-9ddks\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-kube-api-access-9ddks\") pod \"ec94b85a-3d99-42b0-b540-be2d8514c773\" (UID: \"ec94b85a-3d99-42b0-b540-be2d8514c773\") " Nov 01 20:24:33.324418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.324383 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-plugins-config-volume" pod "ec94b85a-3d99-42b0-b540-be2d8514c773" (UID: "ec94b85a-3d99-42b0-b540-be2d8514c773"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:33.324507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.324386 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-routing-overlay-volume" pod "ec94b85a-3d99-42b0-b540-be2d8514c773" (UID: "ec94b85a-3d99-42b0-b540-be2d8514c773"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:33.325167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.325138 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:33.325282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.325171 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec94b85a-3d99-42b0-b540-be2d8514c773-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:33.326763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.326737 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-provider-credentials" pod "ec94b85a-3d99-42b0-b540-be2d8514c773" (UID: "ec94b85a-3d99-42b0-b540-be2d8514c773"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:33.326763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.326750 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-kube-api-access-9ddks" pod "ec94b85a-3d99-42b0-b540-be2d8514c773" (UID: "ec94b85a-3d99-42b0-b540-be2d8514c773"). InnerVolumeSpecName "kube-api-access-9ddks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:33.376168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.376130 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595"] Nov 01 20:24:33.426593 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.426559 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:33.426593 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.426590 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddks\" (UniqueName: \"kubernetes.io/projected/ec94b85a-3d99-42b0-b540-be2d8514c773-kube-api-access-9ddks\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:33.614960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.614872 5633 generic.go:356] "Generic (PLEG): container finished" podID="ec94b85a-3d99-42b0-b540-be2d8514c773" containerID="461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d" exitCode=0 Nov 01 20:24:33.614960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.614922 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" Nov 01 20:24:33.615189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.614960 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" event={"ID":"ec94b85a-3d99-42b0-b540-be2d8514c773","Type":"ContainerDied","Data":"461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d"} Nov 01 20:24:33.615189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.615010 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999" event={"ID":"ec94b85a-3d99-42b0-b540-be2d8514c773","Type":"ContainerDied","Data":"74da876c33914b7543ac7250085edb8bf50ff5d0a1594f41826b1ca5d45cf5f7"} Nov 01 20:24:33.615189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.615040 5633 scope.go:122] "RemoveContainer" containerID="461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d" Nov 01 20:24:33.616732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.616716 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" Nov 01 20:24:33.616810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.616784 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c" event={"ID":"ea602a2d-1c12-4f45-b0e9-ae7809a5820b","Type":"ContainerDied","Data":"dec6217eb1d958d3283c1afdcbe40bf27f2d53e041ed0eacd613d054a66e56d9"} Nov 01 20:24:33.616878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.616839 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dec6217eb1d958d3283c1afdcbe40bf27f2d53e041ed0eacd613d054a66e56d9" Nov 01 20:24:33.619108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.619075 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" event={"ID":"f25fb041-260d-4ec0-86fe-79cef548655f","Type":"ContainerStarted","Data":"cf0b89d5f717d3fd7ac2268e8bc234ae0cc390933a98c1848ddb37bba92ffff6"} Nov 01 20:24:33.619223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.619133 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" event={"ID":"f25fb041-260d-4ec0-86fe-79cef548655f","Type":"ContainerStarted","Data":"bff506a9ff90154058c53de9c0be3e80bc83d7ee05d8d584e612bf7c054a52a1"} Nov 01 20:24:33.629509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.629456 5633 scope.go:122] "RemoveContainer" containerID="461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d" Nov 01 20:24:33.630887 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:24:33.630853 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d\": container with ID starting with 461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d not found: ID does not exist" containerID="461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d" Nov 01 20:24:33.630986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.630898 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d"} err="failed to get container status \"461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d\": rpc error: code = NotFound desc = could not find container \"461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d\": container with ID starting with 461e68595ce4d084c560095ee1475ae123d37d46c36865e23a590d6c9172225d not found: ID does not exist" Nov 01 20:24:33.835075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.835021 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" podStartSLOduration=2.835003805 podStartE2EDuration="2.835003805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:33.83127389 +0000 UTC m=+1281.505533144" watchObservedRunningTime="2026-11-01 20:24:33.835003805 +0000 UTC m=+1281.509263064" Nov 01 20:24:33.944595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.944512 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999"] Nov 01 20:24:33.948588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:33.948565 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-f456pv999"] Nov 01 20:24:34.467424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:34.467392 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec94b85a-3d99-42b0-b540-be2d8514c773" path="/var/lib/kubelet/pods/ec94b85a-3d99-42b0-b540-be2d8514c773/volumes" Nov 01 20:24:34.630345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:34.630295 5633 generic.go:356] "Generic (PLEG): container finished" podID="f25fb041-260d-4ec0-86fe-79cef548655f" containerID="cf0b89d5f717d3fd7ac2268e8bc234ae0cc390933a98c1848ddb37bba92ffff6" exitCode=0 Nov 01 20:24:34.630503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:34.630363 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" event={"ID":"f25fb041-260d-4ec0-86fe-79cef548655f","Type":"ContainerDied","Data":"cf0b89d5f717d3fd7ac2268e8bc234ae0cc390933a98c1848ddb37bba92ffff6"} Nov 01 20:24:35.157714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.157680 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb"] Nov 01 20:24:35.158356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.158313 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ec94b85a-3d99-42b0-b540-be2d8514c773" containerName="payload-processing" Nov 01 20:24:35.158356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.158345 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec94b85a-3d99-42b0-b540-be2d8514c773" containerName="payload-processing" Nov 01 20:24:35.158469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.158391 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ea602a2d-1c12-4f45-b0e9-ae7809a5820b" containerName="revoke-keys" Nov 01 20:24:35.158469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.158397 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea602a2d-1c12-4f45-b0e9-ae7809a5820b" containerName="revoke-keys" Nov 01 20:24:35.158532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.158491 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec94b85a-3d99-42b0-b540-be2d8514c773" containerName="payload-processing" Nov 01 20:24:35.158532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.158504 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea602a2d-1c12-4f45-b0e9-ae7809a5820b" containerName="revoke-keys" Nov 01 20:24:35.162410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.162393 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.218439 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.218386 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb"] Nov 01 20:24:35.250523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.250483 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.250709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.250608 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmx4t\" (UniqueName: \"kubernetes.io/projected/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-kube-api-access-vmx4t\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.250709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.250655 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.351942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.351905 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.352124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.351966 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmx4t\" (UniqueName: \"kubernetes.io/projected/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-kube-api-access-vmx4t\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.352124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.351998 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.352789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.352717 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.354530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.354509 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.409231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.409158 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmx4t\" (UniqueName: \"kubernetes.io/projected/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-kube-api-access-vmx4t\") pod \"maas-api-revoke-sub-481362b9-jsxrb\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.477551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.477510 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:35.790261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.790234 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:35.859482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.859449 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-xks-ca\") pod \"f25fb041-260d-4ec0-86fe-79cef548655f\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " Nov 01 20:24:35.859657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.859595 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25fb041-260d-4ec0-86fe-79cef548655f-kube-api-access-5xd48\" (UniqueName: \"kubernetes.io/projected/f25fb041-260d-4ec0-86fe-79cef548655f-kube-api-access-5xd48\") pod \"f25fb041-260d-4ec0-86fe-79cef548655f\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " Nov 01 20:24:35.859735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.859661 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-openshift-ca\") pod \"f25fb041-260d-4ec0-86fe-79cef548655f\" (UID: \"f25fb041-260d-4ec0-86fe-79cef548655f\") " Nov 01 20:24:35.860196 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.860047 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-openshift-ca" pod "f25fb041-260d-4ec0-86fe-79cef548655f" (UID: "f25fb041-260d-4ec0-86fe-79cef548655f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:35.860275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.860233 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:35.861876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.861849 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25fb041-260d-4ec0-86fe-79cef548655f-kube-api-access-5xd48" pod "f25fb041-260d-4ec0-86fe-79cef548655f" (UID: "f25fb041-260d-4ec0-86fe-79cef548655f"). InnerVolumeSpecName "kube-api-access-5xd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:35.862184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.862164 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-xks-ca" pod "f25fb041-260d-4ec0-86fe-79cef548655f" (UID: "f25fb041-260d-4ec0-86fe-79cef548655f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:35.863713 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:35.863691 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac7d07b_208d_4f79_a1bb_2ca39326acd7.slice/crio-f7357221335e0b9444309f980ca4c730c594ccc7763f2f130bc37cbe8e6a6fc7 WatchSource:0}: Error finding container f7357221335e0b9444309f980ca4c730c594ccc7763f2f130bc37cbe8e6a6fc7: Status 404 returned error can't find the container with id f7357221335e0b9444309f980ca4c730c594ccc7763f2f130bc37cbe8e6a6fc7 Nov 01 20:24:35.864719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.864698 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb"] Nov 01 20:24:35.961597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.961575 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25fb041-260d-4ec0-86fe-79cef548655f-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:35.961750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:35.961602 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xd48\" (UniqueName: \"kubernetes.io/projected/f25fb041-260d-4ec0-86fe-79cef548655f-kube-api-access-5xd48\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:36.047790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.047753 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs"] Nov 01 20:24:36.053818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.053794 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c2aa8df-f56cs"] Nov 01 20:24:36.468205 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.468159 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192caabb-218b-42fa-9b74-d30328130b18" path="/var/lib/kubelet/pods/192caabb-218b-42fa-9b74-d30328130b18/volumes" Nov 01 20:24:36.648288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.648204 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" Nov 01 20:24:36.648288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.648228 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq" event={"ID":"f25fb041-260d-4ec0-86fe-79cef548655f","Type":"ContainerDied","Data":"bff506a9ff90154058c53de9c0be3e80bc83d7ee05d8d584e612bf7c054a52a1"} Nov 01 20:24:36.648769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.648305 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff506a9ff90154058c53de9c0be3e80bc83d7ee05d8d584e612bf7c054a52a1" Nov 01 20:24:36.650076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.650044 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" event={"ID":"4ac7d07b-208d-4f79-a1bb-2ca39326acd7","Type":"ContainerStarted","Data":"8c382f79ae0b3e55d05c98ae92742e22f4ea6633feabe9df21d2e74962d5ea55"} Nov 01 20:24:36.650218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.650096 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" event={"ID":"4ac7d07b-208d-4f79-a1bb-2ca39326acd7","Type":"ContainerStarted","Data":"f7357221335e0b9444309f980ca4c730c594ccc7763f2f130bc37cbe8e6a6fc7"} Nov 01 20:24:36.841355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:36.835693 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" podStartSLOduration=1.8356677540000002 podStartE2EDuration="1.835667754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:36.78961032 +0000 UTC m=+1284.463869602" watchObservedRunningTime="2026-11-01 20:24:36.835667754 +0000 UTC m=+1284.509927015" Nov 01 20:24:37.374382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:37.374350 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:37.528301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:37.528257 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:37.676011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:37.675773 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:37.701372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:37.696455 5633 generic.go:356] "Generic (PLEG): container finished" podID="4ac7d07b-208d-4f79-a1bb-2ca39326acd7" containerID="8c382f79ae0b3e55d05c98ae92742e22f4ea6633feabe9df21d2e74962d5ea55" exitCode=0 Nov 01 20:24:37.701372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:37.696591 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" event={"ID":"4ac7d07b-208d-4f79-a1bb-2ca39326acd7","Type":"ContainerDied","Data":"8c382f79ae0b3e55d05c98ae92742e22f4ea6633feabe9df21d2e74962d5ea55"} Nov 01 20:24:38.379364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:38.378480 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" podUID="db454fc0-1487-43c5-b037-58dc8bbd4071" containerName="payload-processing" containerID="cri-o://02feecc0e56d0c4fbb7f29bbb40d9f6e81d40adf4c1d024b6872ebf433cdff7c" gracePeriod=25 Nov 01 20:24:38.766175 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:38.766137 5633 generic.go:356] "Generic (PLEG): container finished" podID="db454fc0-1487-43c5-b037-58dc8bbd4071" containerID="02feecc0e56d0c4fbb7f29bbb40d9f6e81d40adf4c1d024b6872ebf433cdff7c" exitCode=0 Nov 01 20:24:38.777361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:38.766794 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" event={"ID":"db454fc0-1487-43c5-b037-58dc8bbd4071","Type":"ContainerDied","Data":"02feecc0e56d0c4fbb7f29bbb40d9f6e81d40adf4c1d024b6872ebf433cdff7c"} Nov 01 20:24:39.781627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:39.734513 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:39.827647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:39.822494 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:39.879814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:39.829559 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:40.349681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.346026 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:24:40.349681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.346105 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:40.627546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.627506 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:24:40.627822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.627792 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:40.644012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.642923 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:24:40.644012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.642989 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:40.660380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.649294 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:24:40.660380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.649368 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:40.698188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.698146 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:40.708246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.708203 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:40.711015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.710986 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-kube-api-access-vmx4t\" (UniqueName: \"kubernetes.io/projected/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-kube-api-access-vmx4t\") pod \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " Nov 01 20:24:40.715413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.711351 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-openshift-ca\") pod \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " Nov 01 20:24:40.715870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.715844 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-xks-ca\") pod \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\" (UID: \"4ac7d07b-208d-4f79-a1bb-2ca39326acd7\") " Nov 01 20:24:40.720682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.720653 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-openshift-ca" pod "4ac7d07b-208d-4f79-a1bb-2ca39326acd7" (UID: "4ac7d07b-208d-4f79-a1bb-2ca39326acd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:40.737438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.729076 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-kube-api-access-vmx4t" pod "4ac7d07b-208d-4f79-a1bb-2ca39326acd7" (UID: "4ac7d07b-208d-4f79-a1bb-2ca39326acd7"). InnerVolumeSpecName "kube-api-access-vmx4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:40.737833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.737756 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:40.754383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.745446 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:24:40.776362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.764484 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-xks-ca" pod "4ac7d07b-208d-4f79-a1bb-2ca39326acd7" (UID: "4ac7d07b-208d-4f79-a1bb-2ca39326acd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:40.845359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.844768 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-provider-credentials\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-provider-credentials\") pod \"db454fc0-1487-43c5-b037-58dc8bbd4071\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " Nov 01 20:24:40.845359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.844954 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-plugins-config-volume\") pod \"db454fc0-1487-43c5-b037-58dc8bbd4071\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " Nov 01 20:24:40.845359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.845019 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-routing-overlay-volume\") pod \"db454fc0-1487-43c5-b037-58dc8bbd4071\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " Nov 01 20:24:40.845359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.845151 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-kube-api-access-vnm4k\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-kube-api-access-vnm4k\") pod \"db454fc0-1487-43c5-b037-58dc8bbd4071\" (UID: \"db454fc0-1487-43c5-b037-58dc8bbd4071\") " Nov 01 20:24:40.846055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.845730 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmx4t\" (UniqueName: \"kubernetes.io/projected/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-kube-api-access-vmx4t\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:40.846055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.845753 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:40.846055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.845767 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac7d07b-208d-4f79-a1bb-2ca39326acd7-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:40.853187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.852829 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-plugins-config-volume" pod "db454fc0-1487-43c5-b037-58dc8bbd4071" (UID: "db454fc0-1487-43c5-b037-58dc8bbd4071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:40.858424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.857925 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-provider-credentials" pod "db454fc0-1487-43c5-b037-58dc8bbd4071" (UID: "db454fc0-1487-43c5-b037-58dc8bbd4071"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:40.858424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.858316 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:40.879437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.858837 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:24:40.879437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.869567 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-routing-overlay-volume" pod "db454fc0-1487-43c5-b037-58dc8bbd4071" (UID: "db454fc0-1487-43c5-b037-58dc8bbd4071"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:40.907977 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.907935 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-kube-api-access-vnm4k" pod "db454fc0-1487-43c5-b037-58dc8bbd4071" (UID: "db454fc0-1487-43c5-b037-58dc8bbd4071"). InnerVolumeSpecName "kube-api-access-vnm4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:41.017359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.963357 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:41.017359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.963396 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnm4k\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-kube-api-access-vnm4k\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:41.017359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.963410 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db454fc0-1487-43c5-b037-58dc8bbd4071-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:41.017359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:40.963424 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db454fc0-1487-43c5-b037-58dc8bbd4071-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:41.179359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.178927 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/health\": context deadline exceeded" Nov 01 20:24:41.706948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.706903 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" event={"ID":"4ac7d07b-208d-4f79-a1bb-2ca39326acd7","Type":"ContainerDied","Data":"f7357221335e0b9444309f980ca4c730c594ccc7763f2f130bc37cbe8e6a6fc7"} Nov 01 20:24:41.734989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.734955 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7357221335e0b9444309f980ca4c730c594ccc7763f2f130bc37cbe8e6a6fc7" Nov 01 20:24:41.735274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.735248 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" event={"ID":"db454fc0-1487-43c5-b037-58dc8bbd4071","Type":"ContainerDied","Data":"61411992de70b12a8f9eaaed809b4e47985fe3e11a2bc87047d4e5f324cc6265"} Nov 01 20:24:41.735676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.707318 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb" Nov 01 20:24:41.735937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.725268 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595" Nov 01 20:24:41.736088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.735809 5633 scope.go:122] "RemoveContainer" containerID="02feecc0e56d0c4fbb7f29bbb40d9f6e81d40adf4c1d024b6872ebf433cdff7c" Nov 01 20:24:41.748928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.748056 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 01 20:24:41.831373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:41.827634 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:24:42.116786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.116746 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:42.137697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.137664 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:24:42.231350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.231301 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z"] Nov 01 20:24:42.232504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.232477 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4ac7d07b-208d-4f79-a1bb-2ca39326acd7" containerName="revoke-keys" Nov 01 20:24:42.232663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.232648 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac7d07b-208d-4f79-a1bb-2ca39326acd7" containerName="revoke-keys" Nov 01 20:24:42.232781 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.232770 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f25fb041-260d-4ec0-86fe-79cef548655f" containerName="revoke-keys" Nov 01 20:24:42.232861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.232850 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25fb041-260d-4ec0-86fe-79cef548655f" containerName="revoke-keys" Nov 01 20:24:42.232943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.232934 5633 cpu_manager.go:414] "containerMap: removing container" podUID="db454fc0-1487-43c5-b037-58dc8bbd4071" containerName="payload-processing" Nov 01 20:24:42.233011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.233002 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db454fc0-1487-43c5-b037-58dc8bbd4071" containerName="payload-processing" Nov 01 20:24:42.233212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.233201 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac7d07b-208d-4f79-a1bb-2ca39326acd7" containerName="revoke-keys" Nov 01 20:24:42.233295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.233287 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25fb041-260d-4ec0-86fe-79cef548655f" containerName="revoke-keys" Nov 01 20:24:42.233411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.233401 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="db454fc0-1487-43c5-b037-58dc8bbd4071" containerName="payload-processing" Nov 01 20:24:42.239640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.239616 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.241973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.241908 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z"] Nov 01 20:24:42.262536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.262509 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4"] Nov 01 20:24:42.266444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.266416 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ba28de-sw4l4"] Nov 01 20:24:42.272995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.270882 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595"] Nov 01 20:24:42.275036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.275009 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45k5595"] Nov 01 20:24:42.335539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335498 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7794ca7c-889b-4042-b546-681d2182f466-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335539 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335632 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335703 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7794ca7c-889b-4042-b546-681d2182f466-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335782 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335816 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335889 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt466\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-kube-api-access-nt466\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335924 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.335968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.335950 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437127 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437190 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7794ca7c-889b-4042-b546-681d2182f466-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437221 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437244 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437266 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt466\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-kube-api-access-nt466\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437289 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437322 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437499 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7794ca7c-889b-4042-b546-681d2182f466-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437535 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437683 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.437768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.437733 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.438047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.438025 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.438849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.438067 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.438849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.438452 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7794ca7c-889b-4042-b546-681d2182f466-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.439962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.439941 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.440203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.440185 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7794ca7c-889b-4042-b546-681d2182f466-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.467518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.467488 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949ed387-344f-4683-804f-b4a3052ced10" path="/var/lib/kubelet/pods/949ed387-344f-4683-804f-b4a3052ced10/volumes" Nov 01 20:24:42.467947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.467934 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db454fc0-1487-43c5-b037-58dc8bbd4071" path="/var/lib/kubelet/pods/db454fc0-1487-43c5-b037-58dc8bbd4071/volumes" Nov 01 20:24:42.494729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.494693 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.495437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.495415 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt466\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-kube-api-access-nt466\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.578260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.578226 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:42.738441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.738414 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m"] Nov 01 20:24:42.743794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.743774 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.796316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.796262 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:24:42.796629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.796294 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:24:42.796758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.796714 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:24:42.797242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.797217 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:24:42.799504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.799481 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m"] Nov 01 20:24:42.843203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.843169 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.843518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.843488 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkwc\" (UniqueName: \"kubernetes.io/projected/446c9dd3-226b-46b1-a7e8-3505a15d6dac-kube-api-access-8mkwc\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.843582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.843553 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.944292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.944257 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.944523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.944443 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkwc\" (UniqueName: \"kubernetes.io/projected/446c9dd3-226b-46b1-a7e8-3505a15d6dac-kube-api-access-8mkwc\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.944523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.944475 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.944930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.944904 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:42.947090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:42.947065 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:43.001226 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.001152 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkwc\" (UniqueName: \"kubernetes.io/projected/446c9dd3-226b-46b1-a7e8-3505a15d6dac-kube-api-access-8mkwc\") pod \"maas-api-revoke-sub-e633ffcf-j5r4m\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:43.014217 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:43.014181 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7794ca7c_889b_4042_b546_681d2182f466.slice/crio-3cf3c63fde88f2e9b88949a563975ca6ed4cfdd7c737507275b2930ea3e0ed78 WatchSource:0}: Error finding container 3cf3c63fde88f2e9b88949a563975ca6ed4cfdd7c737507275b2930ea3e0ed78: Status 404 returned error can't find the container with id 3cf3c63fde88f2e9b88949a563975ca6ed4cfdd7c737507275b2930ea3e0ed78 Nov 01 20:24:43.014491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.014470 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z"] Nov 01 20:24:43.016883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.016852 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:24:43.016996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.016926 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:24:43.016996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.016970 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:24:43.058859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.058832 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:43.387007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.386918 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:43.454472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.451907 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m"] Nov 01 20:24:43.745939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.745896 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" event={"ID":"446c9dd3-226b-46b1-a7e8-3505a15d6dac","Type":"ContainerStarted","Data":"bc88f52fef89b554c5ead082f199749363f391c5abac405c404a9b5aadfe9a81"} Nov 01 20:24:43.748176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.748145 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" event={"ID":"7794ca7c-889b-4042-b546-681d2182f466","Type":"ContainerStarted","Data":"7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784"} Nov 01 20:24:43.748359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.748198 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" event={"ID":"7794ca7c-889b-4042-b546-681d2182f466","Type":"ContainerStarted","Data":"3cf3c63fde88f2e9b88949a563975ca6ed4cfdd7c737507275b2930ea3e0ed78"} Nov 01 20:24:43.865954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:43.865894 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podStartSLOduration=2.865875696 podStartE2EDuration="2.865875696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:43.863448647 +0000 UTC m=+1291.537707903" watchObservedRunningTime="2026-11-01 20:24:43.865875696 +0000 UTC m=+1291.540134951" Nov 01 20:24:44.578420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:44.578385 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:44.579871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:44.579843 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 20:24:44.580017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:44.579893 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 20:24:44.760072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:44.760035 5633 generic.go:356] "Generic (PLEG): container finished" podID="446c9dd3-226b-46b1-a7e8-3505a15d6dac" containerID="34e7b34dcb67c0a225678c582b455c399a5bf08161bdafb241b90849314cedcf" exitCode=0 Nov 01 20:24:44.760259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:44.760118 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" event={"ID":"446c9dd3-226b-46b1-a7e8-3505a15d6dac","Type":"ContainerDied","Data":"34e7b34dcb67c0a225678c582b455c399a5bf08161bdafb241b90849314cedcf"} Nov 01 20:24:45.578879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:45.578840 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 20:24:45.579415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:45.578902 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 20:24:45.846490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:45.846405 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c"] Nov 01 20:24:45.851474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:45.851309 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:45.929553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:45.929516 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c"] Nov 01 20:24:45.996365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:45.993705 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:46.001694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.001173 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.001694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.001263 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzzf\" (UniqueName: \"kubernetes.io/projected/a67ca763-e03f-4def-a87c-800b5f434194-kube-api-access-hfzzf\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.001694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.001317 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.107410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.104925 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-xks-ca\") pod \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " Nov 01 20:24:46.107410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.105161 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446c9dd3-226b-46b1-a7e8-3505a15d6dac-kube-api-access-8mkwc\" (UniqueName: \"kubernetes.io/projected/446c9dd3-226b-46b1-a7e8-3505a15d6dac-kube-api-access-8mkwc\") pod \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " Nov 01 20:24:46.107410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.105236 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-openshift-ca\") pod \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\" (UID: \"446c9dd3-226b-46b1-a7e8-3505a15d6dac\") " Nov 01 20:24:46.107410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.105635 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.107410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.105705 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzzf\" (UniqueName: \"kubernetes.io/projected/a67ca763-e03f-4def-a87c-800b5f434194-kube-api-access-hfzzf\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.107410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.105754 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.107410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.106555 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-openshift-ca" pod "446c9dd3-226b-46b1-a7e8-3505a15d6dac" (UID: "446c9dd3-226b-46b1-a7e8-3505a15d6dac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:46.114232 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.108129 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.114694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.114636 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-xks-ca" pod "446c9dd3-226b-46b1-a7e8-3505a15d6dac" (UID: "446c9dd3-226b-46b1-a7e8-3505a15d6dac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:46.117241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.117176 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446c9dd3-226b-46b1-a7e8-3505a15d6dac-kube-api-access-8mkwc" pod "446c9dd3-226b-46b1-a7e8-3505a15d6dac" (UID: "446c9dd3-226b-46b1-a7e8-3505a15d6dac"). InnerVolumeSpecName "kube-api-access-8mkwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:46.117583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.117562 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.163559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.163528 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzzf\" (UniqueName: \"kubernetes.io/projected/a67ca763-e03f-4def-a87c-800b5f434194-kube-api-access-hfzzf\") pod \"maas-api-revoke-sub-d5635f84-rcg8c\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.166667 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:24:46.166626 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:24:46.173777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.173553 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:46.210648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.208432 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkwc\" (UniqueName: \"kubernetes.io/projected/446c9dd3-226b-46b1-a7e8-3505a15d6dac-kube-api-access-8mkwc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:46.210648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.208469 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:46.210648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.208490 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c9dd3-226b-46b1-a7e8-3505a15d6dac-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:46.579325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.579299 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 20:24:46.579629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.579379 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 20:24:46.580156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.580136 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c"] Nov 01 20:24:46.580445 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:24:46.580408 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67ca763_e03f_4def_a87c_800b5f434194.slice/crio-11fe18077a046d2d481b02c07bfe454a04a5447ee734cef37107bbaf09ad0177 WatchSource:0}: Error finding container 11fe18077a046d2d481b02c07bfe454a04a5447ee734cef37107bbaf09ad0177: Status 404 returned error can't find the container with id 11fe18077a046d2d481b02c07bfe454a04a5447ee734cef37107bbaf09ad0177 Nov 01 20:24:46.810536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.810418 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" event={"ID":"a67ca763-e03f-4def-a87c-800b5f434194","Type":"ContainerStarted","Data":"11fe18077a046d2d481b02c07bfe454a04a5447ee734cef37107bbaf09ad0177"} Nov 01 20:24:46.814749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.814709 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" event={"ID":"446c9dd3-226b-46b1-a7e8-3505a15d6dac","Type":"ContainerDied","Data":"bc88f52fef89b554c5ead082f199749363f391c5abac405c404a9b5aadfe9a81"} Nov 01 20:24:46.814895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.814774 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc88f52fef89b554c5ead082f199749363f391c5abac405c404a9b5aadfe9a81" Nov 01 20:24:46.814895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:46.814725 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m" Nov 01 20:24:47.578806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:47.578764 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 20:24:47.578991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:47.578840 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 20:24:47.826068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:47.826037 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" event={"ID":"a67ca763-e03f-4def-a87c-800b5f434194","Type":"ContainerStarted","Data":"c31da941ac225b9572c41ba4e89eb023478b1cd1365a0467b308878d44548ae5"} Nov 01 20:24:48.216251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:48.216215 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:48.579161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:48.579065 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 20:24:48.579161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:48.579122 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 20:24:48.836654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:48.836565 5633 generic.go:356] "Generic (PLEG): container finished" podID="a67ca763-e03f-4def-a87c-800b5f434194" containerID="c31da941ac225b9572c41ba4e89eb023478b1cd1365a0467b308878d44548ae5" exitCode=0 Nov 01 20:24:48.837042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:48.836816 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" event={"ID":"a67ca763-e03f-4def-a87c-800b5f434194","Type":"ContainerDied","Data":"c31da941ac225b9572c41ba4e89eb023478b1cd1365a0467b308878d44548ae5"} Nov 01 20:24:49.057945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.057916 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:49.152165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.152068 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-openshift-ca\") pod \"a67ca763-e03f-4def-a87c-800b5f434194\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " Nov 01 20:24:49.152165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.152128 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67ca763-e03f-4def-a87c-800b5f434194-kube-api-access-hfzzf\" (UniqueName: \"kubernetes.io/projected/a67ca763-e03f-4def-a87c-800b5f434194-kube-api-access-hfzzf\") pod \"a67ca763-e03f-4def-a87c-800b5f434194\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " Nov 01 20:24:49.152472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.152300 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-xks-ca\") pod \"a67ca763-e03f-4def-a87c-800b5f434194\" (UID: \"a67ca763-e03f-4def-a87c-800b5f434194\") " Nov 01 20:24:49.152552 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.152522 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-openshift-ca" pod "a67ca763-e03f-4def-a87c-800b5f434194" (UID: "a67ca763-e03f-4def-a87c-800b5f434194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:49.152951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.152927 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:49.154471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.154441 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67ca763-e03f-4def-a87c-800b5f434194-kube-api-access-hfzzf" pod "a67ca763-e03f-4def-a87c-800b5f434194" (UID: "a67ca763-e03f-4def-a87c-800b5f434194"). InnerVolumeSpecName "kube-api-access-hfzzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:49.154607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.154577 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-xks-ca" pod "a67ca763-e03f-4def-a87c-800b5f434194" (UID: "a67ca763-e03f-4def-a87c-800b5f434194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:49.254043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.254004 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfzzf\" (UniqueName: \"kubernetes.io/projected/a67ca763-e03f-4def-a87c-800b5f434194-kube-api-access-hfzzf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:49.254043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.254044 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67ca763-e03f-4def-a87c-800b5f434194-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:24:49.737998 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.737967 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:49.739017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.738917 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:49.739917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.739895 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:24:49.846554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.846522 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" Nov 01 20:24:49.846947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.846552 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c" event={"ID":"a67ca763-e03f-4def-a87c-800b5f434194","Type":"ContainerDied","Data":"11fe18077a046d2d481b02c07bfe454a04a5447ee734cef37107bbaf09ad0177"} Nov 01 20:24:49.846947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:49.846601 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11fe18077a046d2d481b02c07bfe454a04a5447ee734cef37107bbaf09ad0177" Nov 01 20:24:51.047148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:51.047116 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4"] Nov 01 20:24:51.096383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:51.096348 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755f1583-h6bb4"] Nov 01 20:24:52.469968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:52.469937 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8063796d-dbd8-4656-8e0c-dc2896c7229a" path="/var/lib/kubelet/pods/8063796d-dbd8-4656-8e0c-dc2896c7229a/volumes" Nov 01 20:24:52.877722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:52.877628 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:53.637822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:53.637787 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:24:56.806500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:56.806449 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:57.239359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:57.239116 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:57.515657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:57.514446 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:24:57.810419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:57.809518 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:57.810419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:57.810307 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:24:58.248357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:58.241570 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:58.248357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:58.242391 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:24:58.520553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:58.520454 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:58.611425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:58.611392 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:24:58.689954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:58.689925 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:24:58.728975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:58.728945 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:24:59.083273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:59.082385 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:24:59.117977 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:59.117942 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd"] Nov 01 20:24:59.127752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:59.127718 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c52890b9-t7skd"] Nov 01 20:24:59.288356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:59.282912 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:24:59.765234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:24:59.765100 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:00.059037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.058954 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd"] Nov 01 20:25:00.108460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.108421 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33ac65c-xg9gd"] Nov 01 20:25:00.150911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.150874 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84"] Nov 01 20:25:00.151628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.151605 5633 cpu_manager.go:414] "containerMap: removing container" podUID="446c9dd3-226b-46b1-a7e8-3505a15d6dac" containerName="revoke-keys" Nov 01 20:25:00.151628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.151623 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446c9dd3-226b-46b1-a7e8-3505a15d6dac" containerName="revoke-keys" Nov 01 20:25:00.151840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.151677 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a67ca763-e03f-4def-a87c-800b5f434194" containerName="revoke-keys" Nov 01 20:25:00.151840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.151682 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67ca763-e03f-4def-a87c-800b5f434194" containerName="revoke-keys" Nov 01 20:25:00.151840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.151782 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="446c9dd3-226b-46b1-a7e8-3505a15d6dac" containerName="revoke-keys" Nov 01 20:25:00.151840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.151794 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67ca763-e03f-4def-a87c-800b5f434194" containerName="revoke-keys" Nov 01 20:25:00.155726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.155698 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.210726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.210539 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:25:00.210726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.210648 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:25:00.210995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.210957 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:25:00.213179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.213151 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84"] Nov 01 20:25:00.247184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.247140 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.247384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.247257 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.247384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.247343 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mn6r\" (UniqueName: \"kubernetes.io/projected/9767fef0-2dde-4b52-9cf4-4239566d750e-kube-api-access-6mn6r\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.254821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.254791 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:25:00.349538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.349438 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.349538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.349516 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.349773 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.349605 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mn6r\" (UniqueName: \"kubernetes.io/projected/9767fef0-2dde-4b52-9cf4-4239566d750e-kube-api-access-6mn6r\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.350822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.350730 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.354458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.354407 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.406976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.406934 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mn6r\" (UniqueName: \"kubernetes.io/projected/9767fef0-2dde-4b52-9cf4-4239566d750e-kube-api-access-6mn6r\") pod \"maas-api-revoke-sub-7da88ffc-tgb84\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.469840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.469804 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eff0a38-18e7-4952-8436-ebe3b1653436" path="/var/lib/kubelet/pods/2eff0a38-18e7-4952-8436-ebe3b1653436/volumes" Nov 01 20:25:00.470566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.470548 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540c652b-3449-4fb9-ad3f-6191f42d95f3" path="/var/lib/kubelet/pods/540c652b-3449-4fb9-ad3f-6191f42d95f3/volumes" Nov 01 20:25:00.472685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.472660 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:00.811856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.811821 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d"] Nov 01 20:25:00.817446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.817416 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.856653 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.856721 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.856791 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.856832 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtb8\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-kube-api-access-7rtb8\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.856858 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.856913 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.856953 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b653b1-2d99-4d23-a827-54e4cd53e659-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.857033 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.857388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.857093 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.866467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.866427 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84"] Nov 01 20:25:00.888521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.888489 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71335882-openshift-default-dockercfg-f4t24\"" Nov 01 20:25:00.893162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.893115 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:00.902086 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:25:00.902042 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9767fef0_2dde_4b52_9cf4_4239566d750e.slice/crio-c2532e0ddd9c33e4faea8cadbfd91d041f7459d6d01d5850a7af05aa3fd4ebeb WatchSource:0}: Error finding container c2532e0ddd9c33e4faea8cadbfd91d041f7459d6d01d5850a7af05aa3fd4ebeb: Status 404 returned error can't find the container with id c2532e0ddd9c33e4faea8cadbfd91d041f7459d6d01d5850a7af05aa3fd4ebeb Nov 01 20:25:00.921407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.921366 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d"] Nov 01 20:25:00.964099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.962929 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.964099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.963079 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.979209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.968997 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.979209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.969098 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.982507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.979556 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtb8\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-kube-api-access-7rtb8\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.982507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.979662 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.982507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.979730 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.982507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.979823 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b653b1-2d99-4d23-a827-54e4cd53e659-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.982507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.979949 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.982507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.980565 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:00.983104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.983070 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.001214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:00.983440 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.001842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.001807 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.004285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.004245 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b653b1-2d99-4d23-a827-54e4cd53e659-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.006502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.006432 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.006861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.006816 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.039094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.039044 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtb8\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-kube-api-access-7rtb8\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.039858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.039831 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-mg58d\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.134654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.134551 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:01.218087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.217955 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:01.480499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.480457 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" event={"ID":"9767fef0-2dde-4b52-9cf4-4239566d750e","Type":"ContainerStarted","Data":"44854443a1957b5a5b67dd3a08dd1cfb6453152c73c39f405a1bee1f414552d7"} Nov 01 20:25:01.480659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.480524 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" event={"ID":"9767fef0-2dde-4b52-9cf4-4239566d750e","Type":"ContainerStarted","Data":"c2532e0ddd9c33e4faea8cadbfd91d041f7459d6d01d5850a7af05aa3fd4ebeb"} Nov 01 20:25:01.566992 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:25:01.566961 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b653b1_2d99_4d23_a827_54e4cd53e659.slice/crio-850c70c94b97b9192f8c4284dd0d18c777840076701072299cdd1f730167e1eb WatchSource:0}: Error finding container 850c70c94b97b9192f8c4284dd0d18c777840076701072299cdd1f730167e1eb: Status 404 returned error can't find the container with id 850c70c94b97b9192f8c4284dd0d18c777840076701072299cdd1f730167e1eb Nov 01 20:25:01.567803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.567780 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d"] Nov 01 20:25:01.569998 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.569965 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:25:01.570119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.570041 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:25:01.570119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.570083 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:25:01.594740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:01.594690 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" podStartSLOduration=1.594676333 podStartE2EDuration="1.594676333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:01.592906073 +0000 UTC m=+1309.267165329" watchObservedRunningTime="2026-11-01 20:25:01.594676333 +0000 UTC m=+1309.268935587" Nov 01 20:25:02.490696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:02.490662 5633 generic.go:356] "Generic (PLEG): container finished" podID="9767fef0-2dde-4b52-9cf4-4239566d750e" containerID="44854443a1957b5a5b67dd3a08dd1cfb6453152c73c39f405a1bee1f414552d7" exitCode=0 Nov 01 20:25:02.491166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:02.490746 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" event={"ID":"9767fef0-2dde-4b52-9cf4-4239566d750e","Type":"ContainerDied","Data":"44854443a1957b5a5b67dd3a08dd1cfb6453152c73c39f405a1bee1f414552d7"} Nov 01 20:25:02.492419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:02.492383 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" event={"ID":"81b653b1-2d99-4d23-a827-54e4cd53e659","Type":"ContainerStarted","Data":"972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920"} Nov 01 20:25:02.492530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:02.492443 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" event={"ID":"81b653b1-2d99-4d23-a827-54e4cd53e659","Type":"ContainerStarted","Data":"850c70c94b97b9192f8c4284dd0d18c777840076701072299cdd1f730167e1eb"} Nov 01 20:25:02.722535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:02.722474 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" podStartSLOduration=2.722461255 podStartE2EDuration="2.722461255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:02.718896886 +0000 UTC m=+1310.393156142" watchObservedRunningTime="2026-11-01 20:25:02.722461255 +0000 UTC m=+1310.396720510" Nov 01 20:25:03.135132 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.135096 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:03.140430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.140391 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:03.500718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.500675 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:03.501839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.501815 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:25:03.674134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.674104 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:03.820097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.820008 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-openshift-ca\") pod \"9767fef0-2dde-4b52-9cf4-4239566d750e\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " Nov 01 20:25:03.820097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.820051 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9767fef0-2dde-4b52-9cf4-4239566d750e-kube-api-access-6mn6r\" (UniqueName: \"kubernetes.io/projected/9767fef0-2dde-4b52-9cf4-4239566d750e-kube-api-access-6mn6r\") pod \"9767fef0-2dde-4b52-9cf4-4239566d750e\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " Nov 01 20:25:03.820369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.820131 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-xks-ca\") pod \"9767fef0-2dde-4b52-9cf4-4239566d750e\" (UID: \"9767fef0-2dde-4b52-9cf4-4239566d750e\") " Nov 01 20:25:03.820448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.820393 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-openshift-ca" pod "9767fef0-2dde-4b52-9cf4-4239566d750e" (UID: "9767fef0-2dde-4b52-9cf4-4239566d750e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:03.820748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.820720 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:03.822499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.822472 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-xks-ca" pod "9767fef0-2dde-4b52-9cf4-4239566d750e" (UID: "9767fef0-2dde-4b52-9cf4-4239566d750e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:03.822610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.822597 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9767fef0-2dde-4b52-9cf4-4239566d750e-kube-api-access-6mn6r" pod "9767fef0-2dde-4b52-9cf4-4239566d750e" (UID: "9767fef0-2dde-4b52-9cf4-4239566d750e"). InnerVolumeSpecName "kube-api-access-6mn6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:03.921949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.921911 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mn6r\" (UniqueName: \"kubernetes.io/projected/9767fef0-2dde-4b52-9cf4-4239566d750e-kube-api-access-6mn6r\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:03.921949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:03.921945 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9767fef0-2dde-4b52-9cf4-4239566d750e-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:04.510771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:04.510741 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" Nov 01 20:25:04.511167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:04.510766 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84" event={"ID":"9767fef0-2dde-4b52-9cf4-4239566d750e","Type":"ContainerDied","Data":"c2532e0ddd9c33e4faea8cadbfd91d041f7459d6d01d5850a7af05aa3fd4ebeb"} Nov 01 20:25:04.511167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:04.510827 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2532e0ddd9c33e4faea8cadbfd91d041f7459d6d01d5850a7af05aa3fd4ebeb" Nov 01 20:25:05.308040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:05.308004 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:05.496133 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:05.496077 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:05.635623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:05.635534 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:08.046555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:08.046519 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz"] Nov 01 20:25:08.095681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:08.095650 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56417bbe-4s9wz"] Nov 01 20:25:08.469149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:08.469115 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff23205-cd43-44f2-b390-d3a26d2543ce" path="/var/lib/kubelet/pods/dff23205-cd43-44f2-b390-d3a26d2543ce/volumes" Nov 01 20:25:08.563768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:08.563729 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:08.745669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:08.745566 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:09.413656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:09.413619 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:12.635875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:12.635837 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:25:14.269356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:14.268822 5633 scope.go:122] "RemoveContainer" containerID="f79f5b5f9cfafa7d59ab943374d4bc70692bef3c4a70a3154b14e8a3443d884e" Nov 01 20:25:14.347610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:14.344849 5633 scope.go:122] "RemoveContainer" containerID="31d36dac24597a2a530565760df2986d2fdde791dd264a9dd4991ad51d2e2ff0" Nov 01 20:25:14.417565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:14.417187 5633 scope.go:122] "RemoveContainer" containerID="a8456b8155f1c0354512790cda1928b25175396319983a02e0a3c485c809d223" Nov 01 20:25:14.484532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:14.484506 5633 scope.go:122] "RemoveContainer" containerID="4b455e7405e61b4410f3319179e0b2625da437b3ab8c2f341e2c86e003417d73" Nov 01 20:25:14.646356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:14.637942 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:14.719705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:14.719655 5633 scope.go:122] "RemoveContainer" containerID="5459302c943a880da8770f93aa1224a5c85626f4b9e74696f461427aa8f7a198" Nov 01 20:25:15.127598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:15.113478 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r"] Nov 01 20:25:15.161360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:15.158922 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87c649bc-wf27r"] Nov 01 20:25:16.002674 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.002643 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5"] Nov 01 20:25:16.003276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.003263 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9767fef0-2dde-4b52-9cf4-4239566d750e" containerName="revoke-keys" Nov 01 20:25:16.003343 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.003278 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9767fef0-2dde-4b52-9cf4-4239566d750e" containerName="revoke-keys" Nov 01 20:25:16.003398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.003388 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9767fef0-2dde-4b52-9cf4-4239566d750e" containerName="revoke-keys" Nov 01 20:25:16.009107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.009083 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.049520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.049484 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67"] Nov 01 20:25:16.053538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.053513 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef75212c-fwl67"] Nov 01 20:25:16.065514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.065480 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.065673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.065535 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.065715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.065682 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rm4l\" (UniqueName: \"kubernetes.io/projected/85e94f44-95fe-4306-bff6-69c7176d9f12-kube-api-access-9rm4l\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.094714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.094679 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:25:16.094917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.094751 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:25:16.097238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.097209 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:25:16.097418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.097275 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:25:16.099661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.099629 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5"] Nov 01 20:25:16.165635 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:25:16.165590 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:25:16.167274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.167245 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.167467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.167318 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.167530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.167469 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rm4l\" (UniqueName: \"kubernetes.io/projected/85e94f44-95fe-4306-bff6-69c7176d9f12-kube-api-access-9rm4l\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.167975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.167952 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.169995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.169971 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.224013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.223974 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rm4l\" (UniqueName: \"kubernetes.io/projected/85e94f44-95fe-4306-bff6-69c7176d9f12-kube-api-access-9rm4l\") pod \"maas-api-revoke-sub-6d184280-r2hg5\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.323437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.323326 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:16.471170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.471131 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e93cf36-5d26-4e56-a2ed-a024973b99ed" path="/var/lib/kubelet/pods/9e93cf36-5d26-4e56-a2ed-a024973b99ed/volumes" Nov 01 20:25:16.471868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.471842 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b431aa23-1a34-4995-b07b-015e28965985" path="/var/lib/kubelet/pods/b431aa23-1a34-4995-b07b-015e28965985/volumes" Nov 01 20:25:16.711915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.711887 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5"] Nov 01 20:25:16.713050 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:25:16.713018 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e94f44_95fe_4306_bff6_69c7176d9f12.slice/crio-24704dd53d8dca78d62c89159de27d04740b68637d2d29987c77c864a15fe411 WatchSource:0}: Error finding container 24704dd53d8dca78d62c89159de27d04740b68637d2d29987c77c864a15fe411: Status 404 returned error can't find the container with id 24704dd53d8dca78d62c89159de27d04740b68637d2d29987c77c864a15fe411 Nov 01 20:25:16.727287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:16.727246 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" event={"ID":"85e94f44-95fe-4306-bff6-69c7176d9f12","Type":"ContainerStarted","Data":"24704dd53d8dca78d62c89159de27d04740b68637d2d29987c77c864a15fe411"} Nov 01 20:25:17.061321 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:17.061286 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6"] Nov 01 20:25:17.107560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:17.107521 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f67173f-rvzl6"] Nov 01 20:25:17.738477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:17.738442 5633 generic.go:356] "Generic (PLEG): container finished" podID="85e94f44-95fe-4306-bff6-69c7176d9f12" containerID="881b4285f2258fad151b01c5277313f5d934c40a6498449e171b0a7544af72d9" exitCode=0 Nov 01 20:25:17.738656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:17.738502 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" event={"ID":"85e94f44-95fe-4306-bff6-69c7176d9f12","Type":"ContainerDied","Data":"881b4285f2258fad151b01c5277313f5d934c40a6498449e171b0a7544af72d9"} Nov 01 20:25:18.467900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:18.467863 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84986c63-697f-4e13-9268-2f9398b14229" path="/var/lib/kubelet/pods/84986c63-697f-4e13-9268-2f9398b14229/volumes" Nov 01 20:25:18.901843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:18.901814 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:19.003635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.003533 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-xks-ca\") pod \"85e94f44-95fe-4306-bff6-69c7176d9f12\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " Nov 01 20:25:19.003635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.003636 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e94f44-95fe-4306-bff6-69c7176d9f12-kube-api-access-9rm4l\" (UniqueName: \"kubernetes.io/projected/85e94f44-95fe-4306-bff6-69c7176d9f12-kube-api-access-9rm4l\") pod \"85e94f44-95fe-4306-bff6-69c7176d9f12\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " Nov 01 20:25:19.003856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.003722 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-openshift-ca\") pod \"85e94f44-95fe-4306-bff6-69c7176d9f12\" (UID: \"85e94f44-95fe-4306-bff6-69c7176d9f12\") " Nov 01 20:25:19.004026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.003995 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-openshift-ca" pod "85e94f44-95fe-4306-bff6-69c7176d9f12" (UID: "85e94f44-95fe-4306-bff6-69c7176d9f12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:19.004218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.004201 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:19.005918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.005897 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e94f44-95fe-4306-bff6-69c7176d9f12-kube-api-access-9rm4l" pod "85e94f44-95fe-4306-bff6-69c7176d9f12" (UID: "85e94f44-95fe-4306-bff6-69c7176d9f12"). InnerVolumeSpecName "kube-api-access-9rm4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:19.006025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.005890 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-xks-ca" pod "85e94f44-95fe-4306-bff6-69c7176d9f12" (UID: "85e94f44-95fe-4306-bff6-69c7176d9f12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:19.106090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.106056 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e94f44-95fe-4306-bff6-69c7176d9f12-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:19.106090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.106087 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rm4l\" (UniqueName: \"kubernetes.io/projected/85e94f44-95fe-4306-bff6-69c7176d9f12-kube-api-access-9rm4l\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:19.527853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.527815 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:19.756966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.756935 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" Nov 01 20:25:19.757138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.756974 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5" event={"ID":"85e94f44-95fe-4306-bff6-69c7176d9f12","Type":"ContainerDied","Data":"24704dd53d8dca78d62c89159de27d04740b68637d2d29987c77c864a15fe411"} Nov 01 20:25:19.757138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:19.757023 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24704dd53d8dca78d62c89159de27d04740b68637d2d29987c77c864a15fe411" Nov 01 20:25:24.287378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:24.286587 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:24.876223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:24.876184 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:28.385836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.385801 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t"] Nov 01 20:25:28.386474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.386458 5633 cpu_manager.go:414] "containerMap: removing container" podUID="85e94f44-95fe-4306-bff6-69c7176d9f12" containerName="revoke-keys" Nov 01 20:25:28.386541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.386475 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e94f44-95fe-4306-bff6-69c7176d9f12" containerName="revoke-keys" Nov 01 20:25:28.386604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.386594 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e94f44-95fe-4306-bff6-69c7176d9f12" containerName="revoke-keys" Nov 01 20:25:28.390391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.390371 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.446043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.445999 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t"] Nov 01 20:25:28.489959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.489920 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:25:28.490169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.489984 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:25:28.490169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.490011 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:25:28.490169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.490040 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:25:28.527320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.527264 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.527592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.527571 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.527649 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.527600 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwdq\" (UniqueName: \"kubernetes.io/projected/09712fb1-ca56-4976-9bd5-f83765dfc179-kube-api-access-wdwdq\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.628741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.628705 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.628741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.628747 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwdq\" (UniqueName: \"kubernetes.io/projected/09712fb1-ca56-4976-9bd5-f83765dfc179-kube-api-access-wdwdq\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.629009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.628834 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.629561 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.629535 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.631585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.631563 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.685716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.685649 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwdq\" (UniqueName: \"kubernetes.io/projected/09712fb1-ca56-4976-9bd5-f83765dfc179-kube-api-access-wdwdq\") pod \"maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:28.706098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:28.706069 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:29.096096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:29.096065 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t"] Nov 01 20:25:29.884455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:29.884413 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" event={"ID":"09712fb1-ca56-4976-9bd5-f83765dfc179","Type":"ContainerStarted","Data":"4df4ef295fbd434aadfb03f65ae0ec423f94883b8dc74a6c49b6f1f99b6af6dd"} Nov 01 20:25:29.884862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:29.884480 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" event={"ID":"09712fb1-ca56-4976-9bd5-f83765dfc179","Type":"ContainerStarted","Data":"e2eb67c71282ef40fd6d4e5ffcf84baa38f180719741efaa1b8eb78e24584612"} Nov 01 20:25:29.918273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:29.918233 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:30.005575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:30.005510 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" podStartSLOduration=2.005487149 podStartE2EDuration="2.005487149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:29.996722995 +0000 UTC m=+1337.670982253" watchObservedRunningTime="2026-11-01 20:25:30.005487149 +0000 UTC m=+1337.679746408" Nov 01 20:25:30.895069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:30.895024 5633 generic.go:356] "Generic (PLEG): container finished" podID="09712fb1-ca56-4976-9bd5-f83765dfc179" containerID="4df4ef295fbd434aadfb03f65ae0ec423f94883b8dc74a6c49b6f1f99b6af6dd" exitCode=0 Nov 01 20:25:30.895495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:30.895103 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" event={"ID":"09712fb1-ca56-4976-9bd5-f83765dfc179","Type":"ContainerDied","Data":"4df4ef295fbd434aadfb03f65ae0ec423f94883b8dc74a6c49b6f1f99b6af6dd"} Nov 01 20:25:31.062554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.062514 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7"] Nov 01 20:25:31.066551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.066522 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed4ce23e-n7pm7"] Nov 01 20:25:31.299619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.299579 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn"] Nov 01 20:25:31.304079 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.304053 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.363979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.363944 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn"] Nov 01 20:25:31.469216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.469181 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdf7\" (UniqueName: \"kubernetes.io/projected/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-kube-api-access-9qdf7\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.469440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.469235 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.469440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.469257 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.571096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.570998 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdf7\" (UniqueName: \"kubernetes.io/projected/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-kube-api-access-9qdf7\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.571096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.571048 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.571096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.571069 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.571722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.571697 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.573836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.573813 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.627621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.627591 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdf7\" (UniqueName: \"kubernetes.io/projected/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-kube-api-access-9qdf7\") pod \"maas-api-revoke-sub-4751cbb5-c84pn\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:31.919156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:31.919122 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:32.056797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.055742 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr"] Nov 01 20:25:32.061826 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.061792 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e69b4eb-wz6fr"] Nov 01 20:25:32.092716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.092682 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:32.185099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.185005 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09712fb1-ca56-4976-9bd5-f83765dfc179-kube-api-access-wdwdq\" (UniqueName: \"kubernetes.io/projected/09712fb1-ca56-4976-9bd5-f83765dfc179-kube-api-access-wdwdq\") pod \"09712fb1-ca56-4976-9bd5-f83765dfc179\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " Nov 01 20:25:32.185099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.185084 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-openshift-ca\") pod \"09712fb1-ca56-4976-9bd5-f83765dfc179\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " Nov 01 20:25:32.185297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.185189 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-xks-ca\") pod \"09712fb1-ca56-4976-9bd5-f83765dfc179\" (UID: \"09712fb1-ca56-4976-9bd5-f83765dfc179\") " Nov 01 20:25:32.185572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.185547 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-openshift-ca" pod "09712fb1-ca56-4976-9bd5-f83765dfc179" (UID: "09712fb1-ca56-4976-9bd5-f83765dfc179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:32.185829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.185814 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:32.187465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.187437 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09712fb1-ca56-4976-9bd5-f83765dfc179-kube-api-access-wdwdq" pod "09712fb1-ca56-4976-9bd5-f83765dfc179" (UID: "09712fb1-ca56-4976-9bd5-f83765dfc179"). InnerVolumeSpecName "kube-api-access-wdwdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:32.187465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.187437 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-xks-ca" pod "09712fb1-ca56-4976-9bd5-f83765dfc179" (UID: "09712fb1-ca56-4976-9bd5-f83765dfc179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:32.287598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.287559 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwdq\" (UniqueName: \"kubernetes.io/projected/09712fb1-ca56-4976-9bd5-f83765dfc179-kube-api-access-wdwdq\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:32.287598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.287591 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09712fb1-ca56-4976-9bd5-f83765dfc179-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:32.339480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.339440 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn"] Nov 01 20:25:32.339695 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:25:32.339672 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb4c017_b7d9_4f5d_981e_1596ec5718a5.slice/crio-a1e4be613704a66a1811f7d85e5c0fea30a2da00a00e2c9428746382ea25b9f4 WatchSource:0}: Error finding container a1e4be613704a66a1811f7d85e5c0fea30a2da00a00e2c9428746382ea25b9f4: Status 404 returned error can't find the container with id a1e4be613704a66a1811f7d85e5c0fea30a2da00a00e2c9428746382ea25b9f4 Nov 01 20:25:32.469601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.469570 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1739e3c3-b094-4fc9-bdfb-6b502b372e19" path="/var/lib/kubelet/pods/1739e3c3-b094-4fc9-bdfb-6b502b372e19/volumes" Nov 01 20:25:32.470028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.470012 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0" path="/var/lib/kubelet/pods/c4e18fa9-0d87-4620-9bc4-a97c8a3bd5c0/volumes" Nov 01 20:25:32.914261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.914156 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" event={"ID":"3fb4c017-b7d9-4f5d-981e-1596ec5718a5","Type":"ContainerStarted","Data":"d7b81a76468d69e29074a215b0dd8598d962bccdfce44188ff202572f27f0d0a"} Nov 01 20:25:32.914261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.914217 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" event={"ID":"3fb4c017-b7d9-4f5d-981e-1596ec5718a5","Type":"ContainerStarted","Data":"a1e4be613704a66a1811f7d85e5c0fea30a2da00a00e2c9428746382ea25b9f4"} Nov 01 20:25:32.915655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.915637 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" Nov 01 20:25:32.915767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.915634 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t" event={"ID":"09712fb1-ca56-4976-9bd5-f83765dfc179","Type":"ContainerDied","Data":"e2eb67c71282ef40fd6d4e5ffcf84baa38f180719741efaa1b8eb78e24584612"} Nov 01 20:25:32.915767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:32.915762 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2eb67c71282ef40fd6d4e5ffcf84baa38f180719741efaa1b8eb78e24584612" Nov 01 20:25:33.032615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:33.032548 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" podStartSLOduration=2.032528538 podStartE2EDuration="2.032528538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:33.030975458 +0000 UTC m=+1340.705234713" watchObservedRunningTime="2026-11-01 20:25:33.032528538 +0000 UTC m=+1340.706787794" Nov 01 20:25:33.924868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:33.924834 5633 generic.go:356] "Generic (PLEG): container finished" podID="3fb4c017-b7d9-4f5d-981e-1596ec5718a5" containerID="d7b81a76468d69e29074a215b0dd8598d962bccdfce44188ff202572f27f0d0a" exitCode=0 Nov 01 20:25:33.925056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:33.924915 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" event={"ID":"3fb4c017-b7d9-4f5d-981e-1596ec5718a5","Type":"ContainerDied","Data":"d7b81a76468d69e29074a215b0dd8598d962bccdfce44188ff202572f27f0d0a"} Nov 01 20:25:34.898806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:34.898771 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:35.106926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.106900 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:35.223933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.223902 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-kube-api-access-9qdf7\" (UniqueName: \"kubernetes.io/projected/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-kube-api-access-9qdf7\") pod \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " Nov 01 20:25:35.224128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.224010 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-xks-ca\") pod \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " Nov 01 20:25:35.224276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.224254 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-openshift-ca\") pod \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\" (UID: \"3fb4c017-b7d9-4f5d-981e-1596ec5718a5\") " Nov 01 20:25:35.224745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.224715 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-openshift-ca" pod "3fb4c017-b7d9-4f5d-981e-1596ec5718a5" (UID: "3fb4c017-b7d9-4f5d-981e-1596ec5718a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:35.225098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.225077 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:35.226548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.226525 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-xks-ca" pod "3fb4c017-b7d9-4f5d-981e-1596ec5718a5" (UID: "3fb4c017-b7d9-4f5d-981e-1596ec5718a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:35.226768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.226746 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-kube-api-access-9qdf7" pod "3fb4c017-b7d9-4f5d-981e-1596ec5718a5" (UID: "3fb4c017-b7d9-4f5d-981e-1596ec5718a5"). InnerVolumeSpecName "kube-api-access-9qdf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:35.326779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.326738 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdf7\" (UniqueName: \"kubernetes.io/projected/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-kube-api-access-9qdf7\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:35.326779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.326768 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb4c017-b7d9-4f5d-981e-1596ec5718a5-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:35.943366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.943296 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" event={"ID":"3fb4c017-b7d9-4f5d-981e-1596ec5718a5","Type":"ContainerDied","Data":"a1e4be613704a66a1811f7d85e5c0fea30a2da00a00e2c9428746382ea25b9f4"} Nov 01 20:25:35.943749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.943385 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e4be613704a66a1811f7d85e5c0fea30a2da00a00e2c9428746382ea25b9f4" Nov 01 20:25:35.943749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:35.943309 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn" Nov 01 20:25:39.574097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:39.574066 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:39.949231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:39.949134 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7"] Nov 01 20:25:40.265364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:40.260877 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:40.387181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:40.386816 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:42.703359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:42.695704 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": context deadline exceeded" Nov 01 20:25:44.528397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:44.528355 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:44.698856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:44.698823 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:44.950074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:44.949980 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" podUID="9b3a0214-4207-4963-a48b-c070128bd064" containerName="payload-pre-processing" containerID="cri-o://14a46b60b1802bf190df52d5d43761677667d18969efe8e74beec853a98ff15c" gracePeriod=25 Nov 01 20:25:45.117386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.117320 5633 generic.go:356] "Generic (PLEG): container finished" podID="9b3a0214-4207-4963-a48b-c070128bd064" containerID="14a46b60b1802bf190df52d5d43761677667d18969efe8e74beec853a98ff15c" exitCode=0 Nov 01 20:25:45.117386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.117368 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" event={"ID":"9b3a0214-4207-4963-a48b-c070128bd064","Type":"ContainerDied","Data":"14a46b60b1802bf190df52d5d43761677667d18969efe8e74beec853a98ff15c"} Nov 01 20:25:45.235366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.235319 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:25:45.346240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.346201 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3a0214-4207-4963-a48b-c070128bd064-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3a0214-4207-4963-a48b-c070128bd064-plugins-config-volume\") pod \"9b3a0214-4207-4963-a48b-c070128bd064\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " Nov 01 20:25:45.346534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.346514 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3a0214-4207-4963-a48b-c070128bd064-kube-api-access-7jkhp\" (UniqueName: \"kubernetes.io/projected/9b3a0214-4207-4963-a48b-c070128bd064-kube-api-access-7jkhp\") pod \"9b3a0214-4207-4963-a48b-c070128bd064\" (UID: \"9b3a0214-4207-4963-a48b-c070128bd064\") " Nov 01 20:25:45.346692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.346671 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3a0214-4207-4963-a48b-c070128bd064-plugins-config-volume" pod "9b3a0214-4207-4963-a48b-c070128bd064" (UID: "9b3a0214-4207-4963-a48b-c070128bd064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:45.346969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.346953 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3a0214-4207-4963-a48b-c070128bd064-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:45.349585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.349557 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3a0214-4207-4963-a48b-c070128bd064-kube-api-access-7jkhp" pod "9b3a0214-4207-4963-a48b-c070128bd064" (UID: "9b3a0214-4207-4963-a48b-c070128bd064"). InnerVolumeSpecName "kube-api-access-7jkhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:45.448686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:45.448650 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jkhp\" (UniqueName: \"kubernetes.io/projected/9b3a0214-4207-4963-a48b-c070128bd064-kube-api-access-7jkhp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:46.126452 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.126421 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" Nov 01 20:25:46.126452 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.126428 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7" event={"ID":"9b3a0214-4207-4963-a48b-c070128bd064","Type":"ContainerDied","Data":"4fdb912f250868f61dbeff1cb3d430d2b0b5a243bdc3a6b14493cdb041ed6231"} Nov 01 20:25:46.127041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.126507 5633 scope.go:122] "RemoveContainer" containerID="14a46b60b1802bf190df52d5d43761677667d18969efe8e74beec853a98ff15c" Nov 01 20:25:46.159442 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:25:46.159404 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:25:46.297100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.297061 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7"] Nov 01 20:25:46.344142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.344102 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4e4430-56c9495dbc-8ztj7"] Nov 01 20:25:46.468362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.468299 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3a0214-4207-4963-a48b-c070128bd064" path="/var/lib/kubelet/pods/9b3a0214-4207-4963-a48b-c070128bd064/volumes" Nov 01 20:25:46.624730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.624695 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb"] Nov 01 20:25:46.625363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625315 5633 cpu_manager.go:414] "containerMap: removing container" podUID="09712fb1-ca56-4976-9bd5-f83765dfc179" containerName="revoke-keys" Nov 01 20:25:46.625363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625345 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09712fb1-ca56-4976-9bd5-f83765dfc179" containerName="revoke-keys" Nov 01 20:25:46.625515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625376 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9b3a0214-4207-4963-a48b-c070128bd064" containerName="payload-pre-processing" Nov 01 20:25:46.625515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625381 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3a0214-4207-4963-a48b-c070128bd064" containerName="payload-pre-processing" Nov 01 20:25:46.625515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625404 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3fb4c017-b7d9-4f5d-981e-1596ec5718a5" containerName="revoke-keys" Nov 01 20:25:46.625515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625409 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb4c017-b7d9-4f5d-981e-1596ec5718a5" containerName="revoke-keys" Nov 01 20:25:46.625515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625499 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb4c017-b7d9-4f5d-981e-1596ec5718a5" containerName="revoke-keys" Nov 01 20:25:46.625515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625511 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="09712fb1-ca56-4976-9bd5-f83765dfc179" containerName="revoke-keys" Nov 01 20:25:46.625796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.625520 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3a0214-4207-4963-a48b-c070128bd064" containerName="payload-pre-processing" Nov 01 20:25:46.630696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.630675 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.682912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.682864 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:25:46.682912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.682902 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:25:46.683163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.682947 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:25:46.684633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.684608 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb"] Nov 01 20:25:46.728956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.728862 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:25:46.766632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.766595 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzblt\" (UniqueName: \"kubernetes.io/projected/90f210ad-a9cf-4d57-98b3-06360393d1a4-kube-api-access-nzblt\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.766825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.766640 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.766825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.766757 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.867522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.867488 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.867733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.867664 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzblt\" (UniqueName: \"kubernetes.io/projected/90f210ad-a9cf-4d57-98b3-06360393d1a4-kube-api-access-nzblt\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.867733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.867696 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.868415 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.868386 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.870051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.870027 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.924166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.924137 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzblt\" (UniqueName: \"kubernetes.io/projected/90f210ad-a9cf-4d57-98b3-06360393d1a4-kube-api-access-nzblt\") pod \"maas-api-revoke-sub-29856add-djgqb\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:46.944726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:46.944693 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:47.328897 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:25:47.328867 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f210ad_a9cf_4d57_98b3_06360393d1a4.slice/crio-7c70dc726dd01de220aaf2618de31e111d84381552bf221417f72da32b1b8bcf WatchSource:0}: Error finding container 7c70dc726dd01de220aaf2618de31e111d84381552bf221417f72da32b1b8bcf: Status 404 returned error can't find the container with id 7c70dc726dd01de220aaf2618de31e111d84381552bf221417f72da32b1b8bcf Nov 01 20:25:47.331124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:47.331095 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb"] Nov 01 20:25:48.146707 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:48.146666 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" event={"ID":"90f210ad-a9cf-4d57-98b3-06360393d1a4","Type":"ContainerStarted","Data":"5fe6e40cd9a61e807876affba0801cce9a33a00d698a801aad2e746a96441fb2"} Nov 01 20:25:48.146978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:48.146720 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" event={"ID":"90f210ad-a9cf-4d57-98b3-06360393d1a4","Type":"ContainerStarted","Data":"7c70dc726dd01de220aaf2618de31e111d84381552bf221417f72da32b1b8bcf"} Nov 01 20:25:48.260318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:48.260265 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" podStartSLOduration=2.260251754 podStartE2EDuration="2.260251754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:48.259471032 +0000 UTC m=+1355.933730289" watchObservedRunningTime="2026-11-01 20:25:48.260251754 +0000 UTC m=+1355.934511009" Nov 01 20:25:49.158233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:49.158192 5633 generic.go:356] "Generic (PLEG): container finished" podID="90f210ad-a9cf-4d57-98b3-06360393d1a4" containerID="5fe6e40cd9a61e807876affba0801cce9a33a00d698a801aad2e746a96441fb2" exitCode=0 Nov 01 20:25:49.158662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:49.158281 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" event={"ID":"90f210ad-a9cf-4d57-98b3-06360393d1a4","Type":"ContainerDied","Data":"5fe6e40cd9a61e807876affba0801cce9a33a00d698a801aad2e746a96441fb2"} Nov 01 20:25:49.395496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:49.395456 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:50.325763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.325732 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:50.414215 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.414180 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-openshift-ca\") pod \"90f210ad-a9cf-4d57-98b3-06360393d1a4\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " Nov 01 20:25:50.414429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.414378 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f210ad-a9cf-4d57-98b3-06360393d1a4-kube-api-access-nzblt\" (UniqueName: \"kubernetes.io/projected/90f210ad-a9cf-4d57-98b3-06360393d1a4-kube-api-access-nzblt\") pod \"90f210ad-a9cf-4d57-98b3-06360393d1a4\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " Nov 01 20:25:50.414507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.414474 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-xks-ca\") pod \"90f210ad-a9cf-4d57-98b3-06360393d1a4\" (UID: \"90f210ad-a9cf-4d57-98b3-06360393d1a4\") " Nov 01 20:25:50.414592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.414566 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-openshift-ca" pod "90f210ad-a9cf-4d57-98b3-06360393d1a4" (UID: "90f210ad-a9cf-4d57-98b3-06360393d1a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:50.414996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.414976 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:50.416715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.416688 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-xks-ca" pod "90f210ad-a9cf-4d57-98b3-06360393d1a4" (UID: "90f210ad-a9cf-4d57-98b3-06360393d1a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:50.416908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.416883 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f210ad-a9cf-4d57-98b3-06360393d1a4-kube-api-access-nzblt" pod "90f210ad-a9cf-4d57-98b3-06360393d1a4" (UID: "90f210ad-a9cf-4d57-98b3-06360393d1a4"). InnerVolumeSpecName "kube-api-access-nzblt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:50.516456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.516418 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzblt\" (UniqueName: \"kubernetes.io/projected/90f210ad-a9cf-4d57-98b3-06360393d1a4-kube-api-access-nzblt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:50.516456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:50.516450 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f210ad-a9cf-4d57-98b3-06360393d1a4-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:25:51.178681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:51.178608 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" Nov 01 20:25:51.178858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:51.178607 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb" event={"ID":"90f210ad-a9cf-4d57-98b3-06360393d1a4","Type":"ContainerDied","Data":"7c70dc726dd01de220aaf2618de31e111d84381552bf221417f72da32b1b8bcf"} Nov 01 20:25:51.178858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:51.178771 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c70dc726dd01de220aaf2618de31e111d84381552bf221417f72da32b1b8bcf" Nov 01 20:25:54.111296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:54.111260 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:25:57.550554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:57.550497 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:25:58.970123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:25:58.970019 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:00.701784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:00.701734 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:26:00.793883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:00.793473 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:26:03.432286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:03.432242 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:04.969136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:04.969093 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:05.267007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:05.266905 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:05.423428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:05.423387 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:05.476864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:05.476826 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t"] Nov 01 20:26:05.528814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:05.528727 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a4e4430-g6v6t"] Nov 01 20:26:05.581733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:05.581697 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:06.172629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:06.172582 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:06.313186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:06.313143 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:06.469570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:06.469531 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09712fb1-ca56-4976-9bd5-f83765dfc179" path="/var/lib/kubelet/pods/09712fb1-ca56-4976-9bd5-f83765dfc179/volumes" Nov 01 20:26:06.478545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:06.478508 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:07.107996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:07.103240 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:07.313126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:07.313076 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:09.476265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:09.476231 5633 generic.go:356] "Generic (PLEG): container finished" podID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerID="972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920" exitCode=0 Nov 01 20:26:09.476770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:09.476304 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" event={"ID":"81b653b1-2d99-4d23-a827-54e4cd53e659","Type":"ContainerDied","Data":"972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920"} Nov 01 20:26:09.476770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:09.476731 5633 scope.go:122] "RemoveContainer" containerID="972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920" Nov 01 20:26:09.478254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:09.478226 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:26:09.478394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:09.478293 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:26:09.478394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:09.478325 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:26:10.118544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.117863 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz"] Nov 01 20:26:10.121363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.119758 5633 cpu_manager.go:414] "containerMap: removing container" podUID="90f210ad-a9cf-4d57-98b3-06360393d1a4" containerName="revoke-keys" Nov 01 20:26:10.121363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.119788 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f210ad-a9cf-4d57-98b3-06360393d1a4" containerName="revoke-keys" Nov 01 20:26:10.121363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.120149 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f210ad-a9cf-4d57-98b3-06360393d1a4" containerName="revoke-keys" Nov 01 20:26:10.128288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.128252 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.161431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.161399 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz"] Nov 01 20:26:10.180530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.180490 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:26:10.180530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.180526 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:26:10.180805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.180603 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:26:10.180805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.180766 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:26:10.229199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.229161 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.229434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.229413 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.229497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.229461 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcdf\" (UniqueName: \"kubernetes.io/projected/8ab51c6f-cf5e-4397-805f-70d307fc6105-kube-api-access-tvcdf\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.331129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.331093 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.331298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.331141 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcdf\" (UniqueName: \"kubernetes.io/projected/8ab51c6f-cf5e-4397-805f-70d307fc6105-kube-api-access-tvcdf\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.331298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.331214 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.331883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.331850 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.333690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.333671 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.412211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.412127 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcdf\" (UniqueName: \"kubernetes.io/projected/8ab51c6f-cf5e-4397-805f-70d307fc6105-kube-api-access-tvcdf\") pod \"maas-api-revoke-sub-e5ebfb90-6hhpz\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.442642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.442606 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:10.518889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.518851 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" event={"ID":"81b653b1-2d99-4d23-a827-54e4cd53e659","Type":"ContainerStarted","Data":"fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653"} Nov 01 20:26:10.558157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.558053 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:10.757914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.757862 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:10.853636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:10.853404 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz"] Nov 01 20:26:10.854062 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:26:10.854034 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab51c6f_cf5e_4397_805f_70d307fc6105.slice/crio-1db4343a0070125870c913e6eaf57bf8f5e0d4b84b569cbe2dc6bec38532ddf2 WatchSource:0}: Error finding container 1db4343a0070125870c913e6eaf57bf8f5e0d4b84b569cbe2dc6bec38532ddf2: Status 404 returned error can't find the container with id 1db4343a0070125870c913e6eaf57bf8f5e0d4b84b569cbe2dc6bec38532ddf2 Nov 01 20:26:11.008432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:11.007937 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:11.134978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:11.134942 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:26:11.135162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:11.135130 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 20:26:11.135218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:11.135174 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 20:26:11.529787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:11.529742 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" event={"ID":"8ab51c6f-cf5e-4397-805f-70d307fc6105","Type":"ContainerStarted","Data":"b28559eeb41cb2e407df29502921e6aefcff8c1a246b6ce3583fa98794d7c346"} Nov 01 20:26:11.530276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:11.529820 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" event={"ID":"8ab51c6f-cf5e-4397-805f-70d307fc6105","Type":"ContainerStarted","Data":"1db4343a0070125870c913e6eaf57bf8f5e0d4b84b569cbe2dc6bec38532ddf2"} Nov 01 20:26:11.648044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:11.647991 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" podStartSLOduration=1.6479767079999998 podStartE2EDuration="1.647976708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:11.643995684 +0000 UTC m=+1379.318254940" watchObservedRunningTime="2026-11-01 20:26:11.647976708 +0000 UTC m=+1379.322235962" Nov 01 20:26:12.135570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:12.135538 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 20:26:12.135755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:12.135592 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 20:26:12.544235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:12.544189 5633 generic.go:356] "Generic (PLEG): container finished" podID="8ab51c6f-cf5e-4397-805f-70d307fc6105" containerID="b28559eeb41cb2e407df29502921e6aefcff8c1a246b6ce3583fa98794d7c346" exitCode=0 Nov 01 20:26:12.544762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:12.544263 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" event={"ID":"8ab51c6f-cf5e-4397-805f-70d307fc6105","Type":"ContainerDied","Data":"b28559eeb41cb2e407df29502921e6aefcff8c1a246b6ce3583fa98794d7c346"} Nov 01 20:26:13.135719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:13.135680 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 20:26:13.135899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:13.135747 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 20:26:13.350878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:13.350848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:13.539359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:13.539052 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:14.030136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.030103 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:14.136019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.135935 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 20:26:14.136019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.135995 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 20:26:14.136019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.136011 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab51c6f-cf5e-4397-805f-70d307fc6105-kube-api-access-tvcdf\" (UniqueName: \"kubernetes.io/projected/8ab51c6f-cf5e-4397-805f-70d307fc6105-kube-api-access-tvcdf\") pod \"8ab51c6f-cf5e-4397-805f-70d307fc6105\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " Nov 01 20:26:14.136243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.136172 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-openshift-ca\") pod \"8ab51c6f-cf5e-4397-805f-70d307fc6105\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " Nov 01 20:26:14.136243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.136211 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-xks-ca\") pod \"8ab51c6f-cf5e-4397-805f-70d307fc6105\" (UID: \"8ab51c6f-cf5e-4397-805f-70d307fc6105\") " Nov 01 20:26:14.136595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.136567 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-openshift-ca" pod "8ab51c6f-cf5e-4397-805f-70d307fc6105" (UID: "8ab51c6f-cf5e-4397-805f-70d307fc6105"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:14.136838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.136777 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:14.138457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.138431 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab51c6f-cf5e-4397-805f-70d307fc6105-kube-api-access-tvcdf" pod "8ab51c6f-cf5e-4397-805f-70d307fc6105" (UID: "8ab51c6f-cf5e-4397-805f-70d307fc6105"). InnerVolumeSpecName "kube-api-access-tvcdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:14.138566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.138521 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-xks-ca" pod "8ab51c6f-cf5e-4397-805f-70d307fc6105" (UID: "8ab51c6f-cf5e-4397-805f-70d307fc6105"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:14.237823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.237776 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab51c6f-cf5e-4397-805f-70d307fc6105-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:14.237823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.237810 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcdf\" (UniqueName: \"kubernetes.io/projected/8ab51c6f-cf5e-4397-805f-70d307fc6105-kube-api-access-tvcdf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:14.611050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.611010 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" Nov 01 20:26:14.611399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.611041 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz" event={"ID":"8ab51c6f-cf5e-4397-805f-70d307fc6105","Type":"ContainerDied","Data":"1db4343a0070125870c913e6eaf57bf8f5e0d4b84b569cbe2dc6bec38532ddf2"} Nov 01 20:26:14.611520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:14.611419 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db4343a0070125870c913e6eaf57bf8f5e0d4b84b569cbe2dc6bec38532ddf2" Nov 01 20:26:15.135072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.135037 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 20:26:15.135485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.135105 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 20:26:15.254086 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.254051 5633 scope.go:122] "RemoveContainer" containerID="c6dee954ae2fcd70a66429802d7baa00e9d3d64d3b06c775dd573743973b9f8a" Nov 01 20:26:15.264951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.264932 5633 scope.go:122] "RemoveContainer" containerID="5e3a8777f12664f87440622c9d5d925d33d47e7a9936589c88adb622f9fb6755" Nov 01 20:26:15.276166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.276139 5633 scope.go:122] "RemoveContainer" containerID="8e239d8bde978e861716ad42f104c97ba2618bc41e5a2d3b40e198a2412a08d0" Nov 01 20:26:15.312046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.312019 5633 scope.go:122] "RemoveContainer" containerID="baa837faa6efa6a6c39e59d07c896bdddb5b7fbfe373bb875920e597b7017940" Nov 01 20:26:15.323568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.323549 5633 scope.go:122] "RemoveContainer" containerID="07a98b8fec983208f50f2fb68e2f5f53452aa447a9ffe1cc910e97f0ecdc5b05" Nov 01 20:26:15.334145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.334125 5633 scope.go:122] "RemoveContainer" containerID="8eb69e5438f6fe3a6e541d620b7d3660b8a5cf1c561d672998007b7191a87e70" Nov 01 20:26:15.344364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.344319 5633 scope.go:122] "RemoveContainer" containerID="26fa9da03abc2caa9e808e0b55830e774c3015cd02773f13b4d0f3739759f5d5" Nov 01 20:26:15.354916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.354891 5633 scope.go:122] "RemoveContainer" containerID="afd9cf34930e73c75da5b003c42cf4521707cc79d0cd0ebed6d734c07aa84f3d" Nov 01 20:26:15.369520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.369499 5633 scope.go:122] "RemoveContainer" containerID="163c66c68b96f27f7a30cfbc8ddc1e3a5521f43541adcbd7f077ba92a9b861b9" Nov 01 20:26:15.381791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.381772 5633 scope.go:122] "RemoveContainer" containerID="927a942df956068bb8290c2eb2c001d5045bd14806b6016fb76bac10dd58f254" Nov 01 20:26:15.430600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.429220 5633 scope.go:122] "RemoveContainer" containerID="dae33e3ff354471f68bd06078087aa45d8adf491b652a1e437760920cc1b84a7" Nov 01 20:26:15.811737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:15.811690 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:16.139840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:16.139754 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:26:16.140386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:16.140068 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:26:16.141053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:16.141026 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:26:16.163639 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:26:16.163594 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:26:17.262857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:17.262827 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:26:19.051084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:19.051041 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5"] Nov 01 20:26:19.100301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:19.100267 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0342a788-zm7z5"] Nov 01 20:26:20.546356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:20.525984 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf" path="/var/lib/kubelet/pods/b9434a3e-0ec3-4dd5-bb4c-d3c526b29dbf/volumes" Nov 01 20:26:21.416540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:21.416499 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:25.213162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.213118 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:25.340078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.340038 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:25.456987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.456952 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2"] Nov 01 20:26:25.457623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.457608 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8ab51c6f-cf5e-4397-805f-70d307fc6105" containerName="revoke-keys" Nov 01 20:26:25.457682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.457625 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab51c6f-cf5e-4397-805f-70d307fc6105" containerName="revoke-keys" Nov 01 20:26:25.457756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.457746 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab51c6f-cf5e-4397-805f-70d307fc6105" containerName="revoke-keys" Nov 01 20:26:25.463577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.463525 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.516648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.516616 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2"] Nov 01 20:26:25.560772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.560731 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:26:25.560987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.560792 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:26:25.560987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.560880 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:26:25.564577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.564552 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:26:25.618496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.618466 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qdhp\" (UniqueName: \"kubernetes.io/projected/0ff429fe-cecd-4ab4-8155-de32c795f4fc-kube-api-access-8qdhp\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.618667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.618541 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.618742 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.618723 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.719917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.719827 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.719917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.719909 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdhp\" (UniqueName: \"kubernetes.io/projected/0ff429fe-cecd-4ab4-8155-de32c795f4fc-kube-api-access-8qdhp\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.720173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.719975 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.720835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.720693 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.722594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.722572 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.776434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.776395 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdhp\" (UniqueName: \"kubernetes.io/projected/0ff429fe-cecd-4ab4-8155-de32c795f4fc-kube-api-access-8qdhp\") pod \"maas-api-revoke-sub-5e28c32c-dpfq2\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:25.779203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:25.779175 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:26.080625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.080533 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:26.167398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.162813 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2"] Nov 01 20:26:26.360585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.360505 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct"] Nov 01 20:26:26.366282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.366261 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.444008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.443963 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct"] Nov 01 20:26:26.532442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.532412 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.532651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.532468 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.532651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.532643 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn46q\" (UniqueName: \"kubernetes.io/projected/a4415b8c-2489-4872-a47b-56ffbaa1a212-kube-api-access-pn46q\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.633738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.633662 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.633876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.633742 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn46q\" (UniqueName: \"kubernetes.io/projected/a4415b8c-2489-4872-a47b-56ffbaa1a212-kube-api-access-pn46q\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.634006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.633988 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.634409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.634389 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.636433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.636416 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.689941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.689904 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn46q\" (UniqueName: \"kubernetes.io/projected/a4415b8c-2489-4872-a47b-56ffbaa1a212-kube-api-access-pn46q\") pod \"maas-api-revoke-sub-d05e489c-sqhct\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:26.855462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.855429 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" event={"ID":"0ff429fe-cecd-4ab4-8155-de32c795f4fc","Type":"ContainerStarted","Data":"9834fbaeb0b3bcdfc463731208ee883ca6c945a6575e1a5d0315c7da700cd128"} Nov 01 20:26:26.855656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.855482 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" event={"ID":"0ff429fe-cecd-4ab4-8155-de32c795f4fc","Type":"ContainerStarted","Data":"ace155c975f8cf823c4b13c6d929d61ce01d593348c775714ba580070d139159"} Nov 01 20:26:26.970740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.970682 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" podStartSLOduration=1.9706636180000001 podStartE2EDuration="1.970663618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:26.966654547 +0000 UTC m=+1394.640913805" watchObservedRunningTime="2026-11-01 20:26:26.970663618 +0000 UTC m=+1394.644922875" Nov 01 20:26:26.980429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:26.980390 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:27.357779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:27.357749 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct"] Nov 01 20:26:27.357962 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:26:27.357942 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4415b8c_2489_4872_a47b_56ffbaa1a212.slice/crio-8ee07a792728208029f47c809cae83e5109e6bc65540e48192d316b827a6f6cf WatchSource:0}: Error finding container 8ee07a792728208029f47c809cae83e5109e6bc65540e48192d316b827a6f6cf: Status 404 returned error can't find the container with id 8ee07a792728208029f47c809cae83e5109e6bc65540e48192d316b827a6f6cf Nov 01 20:26:27.866108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:27.865996 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" event={"ID":"a4415b8c-2489-4872-a47b-56ffbaa1a212","Type":"ContainerStarted","Data":"26c2c144e1af38915c60086315a7540042a3e81b6e2409544827819969d1ca7a"} Nov 01 20:26:27.866108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:27.866057 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" event={"ID":"a4415b8c-2489-4872-a47b-56ffbaa1a212","Type":"ContainerStarted","Data":"8ee07a792728208029f47c809cae83e5109e6bc65540e48192d316b827a6f6cf"} Nov 01 20:26:27.868060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:27.868033 5633 generic.go:356] "Generic (PLEG): container finished" podID="0ff429fe-cecd-4ab4-8155-de32c795f4fc" containerID="9834fbaeb0b3bcdfc463731208ee883ca6c945a6575e1a5d0315c7da700cd128" exitCode=0 Nov 01 20:26:27.868178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:27.868071 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" event={"ID":"0ff429fe-cecd-4ab4-8155-de32c795f4fc","Type":"ContainerDied","Data":"9834fbaeb0b3bcdfc463731208ee883ca6c945a6575e1a5d0315c7da700cd128"} Nov 01 20:26:27.983021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:27.982961 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" podStartSLOduration=1.982948114 podStartE2EDuration="1.982948114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:27.978454879 +0000 UTC m=+1395.652714135" watchObservedRunningTime="2026-11-01 20:26:27.982948114 +0000 UTC m=+1395.657207368" Nov 01 20:26:28.877813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:28.877776 5633 generic.go:356] "Generic (PLEG): container finished" podID="a4415b8c-2489-4872-a47b-56ffbaa1a212" containerID="26c2c144e1af38915c60086315a7540042a3e81b6e2409544827819969d1ca7a" exitCode=0 Nov 01 20:26:28.878296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:28.877861 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" event={"ID":"a4415b8c-2489-4872-a47b-56ffbaa1a212","Type":"ContainerDied","Data":"26c2c144e1af38915c60086315a7540042a3e81b6e2409544827819969d1ca7a"} Nov 01 20:26:29.046902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.046875 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:29.167153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.167065 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-xks-ca\") pod \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " Nov 01 20:26:29.167153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.167139 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-openshift-ca\") pod \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " Nov 01 20:26:29.167522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.167498 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff429fe-cecd-4ab4-8155-de32c795f4fc-kube-api-access-8qdhp\" (UniqueName: \"kubernetes.io/projected/0ff429fe-cecd-4ab4-8155-de32c795f4fc-kube-api-access-8qdhp\") pod \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\" (UID: \"0ff429fe-cecd-4ab4-8155-de32c795f4fc\") " Nov 01 20:26:29.168264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.167663 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-openshift-ca" pod "0ff429fe-cecd-4ab4-8155-de32c795f4fc" (UID: "0ff429fe-cecd-4ab4-8155-de32c795f4fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:29.168804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.168783 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:29.170601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.170523 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-xks-ca" pod "0ff429fe-cecd-4ab4-8155-de32c795f4fc" (UID: "0ff429fe-cecd-4ab4-8155-de32c795f4fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:29.172285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.172249 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff429fe-cecd-4ab4-8155-de32c795f4fc-kube-api-access-8qdhp" pod "0ff429fe-cecd-4ab4-8155-de32c795f4fc" (UID: "0ff429fe-cecd-4ab4-8155-de32c795f4fc"). InnerVolumeSpecName "kube-api-access-8qdhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:29.270650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.270615 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qdhp\" (UniqueName: \"kubernetes.io/projected/0ff429fe-cecd-4ab4-8155-de32c795f4fc-kube-api-access-8qdhp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:29.270650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.270647 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff429fe-cecd-4ab4-8155-de32c795f4fc-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:29.887221 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.887198 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" Nov 01 20:26:29.887678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.887192 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2" event={"ID":"0ff429fe-cecd-4ab4-8155-de32c795f4fc","Type":"ContainerDied","Data":"ace155c975f8cf823c4b13c6d929d61ce01d593348c775714ba580070d139159"} Nov 01 20:26:29.887678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:29.887314 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace155c975f8cf823c4b13c6d929d61ce01d593348c775714ba580070d139159" Nov 01 20:26:30.108416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.084572 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:30.108416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.086017 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-xks-ca\") pod \"a4415b8c-2489-4872-a47b-56ffbaa1a212\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " Nov 01 20:26:30.108416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.086190 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4415b8c-2489-4872-a47b-56ffbaa1a212-kube-api-access-pn46q\" (UniqueName: \"kubernetes.io/projected/a4415b8c-2489-4872-a47b-56ffbaa1a212-kube-api-access-pn46q\") pod \"a4415b8c-2489-4872-a47b-56ffbaa1a212\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " Nov 01 20:26:30.108416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.086254 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-openshift-ca\") pod \"a4415b8c-2489-4872-a47b-56ffbaa1a212\" (UID: \"a4415b8c-2489-4872-a47b-56ffbaa1a212\") " Nov 01 20:26:30.108416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.092557 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-openshift-ca" pod "a4415b8c-2489-4872-a47b-56ffbaa1a212" (UID: "a4415b8c-2489-4872-a47b-56ffbaa1a212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:30.113696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.113629 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-xks-ca" pod "a4415b8c-2489-4872-a47b-56ffbaa1a212" (UID: "a4415b8c-2489-4872-a47b-56ffbaa1a212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:30.115019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.114947 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:30.115019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.114978 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4415b8c-2489-4872-a47b-56ffbaa1a212-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:30.118070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.118039 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4415b8c-2489-4872-a47b-56ffbaa1a212-kube-api-access-pn46q" pod "a4415b8c-2489-4872-a47b-56ffbaa1a212" (UID: "a4415b8c-2489-4872-a47b-56ffbaa1a212"). InnerVolumeSpecName "kube-api-access-pn46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:30.215941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.215901 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn46q\" (UniqueName: \"kubernetes.io/projected/a4415b8c-2489-4872-a47b-56ffbaa1a212-kube-api-access-pn46q\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:30.876994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.876913 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:30.902298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.902249 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" event={"ID":"a4415b8c-2489-4872-a47b-56ffbaa1a212","Type":"ContainerDied","Data":"8ee07a792728208029f47c809cae83e5109e6bc65540e48192d316b827a6f6cf"} Nov 01 20:26:30.902298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.902386 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ee07a792728208029f47c809cae83e5109e6bc65540e48192d316b827a6f6cf" Nov 01 20:26:30.902946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:30.902514 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct" Nov 01 20:26:31.762117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:31.762074 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:34.879390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:34.879345 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:35.046108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:35.046075 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l"] Nov 01 20:26:35.097751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:35.097712 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce44211-jd66l"] Nov 01 20:26:36.109358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:36.089168 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f"] Nov 01 20:26:36.183832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:36.183651 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-243acbce-7vz2f"] Nov 01 20:26:36.540492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:36.540454 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8861be-1d5f-4567-9618-015efeddaef3" path="/var/lib/kubelet/pods/3c8861be-1d5f-4567-9618-015efeddaef3/volumes" Nov 01 20:26:36.541489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:36.541463 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9d59d2-d4be-4142-bf51-1e7c4ed9217b" path="/var/lib/kubelet/pods/cc9d59d2-d4be-4142-bf51-1e7c4ed9217b/volumes" Nov 01 20:26:36.549066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:36.549034 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:37.064857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.064822 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4"] Nov 01 20:26:37.065578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.065558 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0ff429fe-cecd-4ab4-8155-de32c795f4fc" containerName="revoke-keys" Nov 01 20:26:37.065578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.065579 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff429fe-cecd-4ab4-8155-de32c795f4fc" containerName="revoke-keys" Nov 01 20:26:37.065775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.065632 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a4415b8c-2489-4872-a47b-56ffbaa1a212" containerName="revoke-keys" Nov 01 20:26:37.065775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.065640 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4415b8c-2489-4872-a47b-56ffbaa1a212" containerName="revoke-keys" Nov 01 20:26:37.065884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.065791 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff429fe-cecd-4ab4-8155-de32c795f4fc" containerName="revoke-keys" Nov 01 20:26:37.065884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.065816 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4415b8c-2489-4872-a47b-56ffbaa1a212" containerName="revoke-keys" Nov 01 20:26:37.070813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.070791 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.125638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.125600 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4"] Nov 01 20:26:37.168661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.168625 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:26:37.168661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.168633 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:26:37.168661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.168658 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:26:37.171980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.171956 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:26:37.222077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.222033 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.222265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.222247 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.222434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.222415 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s84\" (UniqueName: \"kubernetes.io/projected/1eb6f177-e8fa-43ab-878f-4ba23da92b07-kube-api-access-h5s84\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.323945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.323826 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.324176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.324046 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.324176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.324162 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s84\" (UniqueName: \"kubernetes.io/projected/1eb6f177-e8fa-43ab-878f-4ba23da92b07-kube-api-access-h5s84\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.324996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.324969 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.326737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.326716 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.380269 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.380239 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s84\" (UniqueName: \"kubernetes.io/projected/1eb6f177-e8fa-43ab-878f-4ba23da92b07-kube-api-access-h5s84\") pod \"maas-api-revoke-sub-e49108ba-lzhk4\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.389267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.389239 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:37.769237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:37.769204 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4"] Nov 01 20:26:37.769559 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:26:37.769535 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb6f177_e8fa_43ab_878f_4ba23da92b07.slice/crio-6d7231da7137455cf21d03dcc4cb8853681884392917b1d50332e7cab07222f1 WatchSource:0}: Error finding container 6d7231da7137455cf21d03dcc4cb8853681884392917b1d50332e7cab07222f1: Status 404 returned error can't find the container with id 6d7231da7137455cf21d03dcc4cb8853681884392917b1d50332e7cab07222f1 Nov 01 20:26:38.122249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:38.122208 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" event={"ID":"1eb6f177-e8fa-43ab-878f-4ba23da92b07","Type":"ContainerStarted","Data":"a46e2670a81d69ebdba283cb2c37eeee84ef6189a48fb163e1741062d28000d3"} Nov 01 20:26:38.122417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:38.122280 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" event={"ID":"1eb6f177-e8fa-43ab-878f-4ba23da92b07","Type":"ContainerStarted","Data":"6d7231da7137455cf21d03dcc4cb8853681884392917b1d50332e7cab07222f1"} Nov 01 20:26:38.235842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:38.235773 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" podStartSLOduration=1.235753364 podStartE2EDuration="1.235753364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:38.234136292 +0000 UTC m=+1405.908395547" watchObservedRunningTime="2026-11-01 20:26:38.235753364 +0000 UTC m=+1405.910012619" Nov 01 20:26:39.131906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.131812 5633 generic.go:356] "Generic (PLEG): container finished" podID="1eb6f177-e8fa-43ab-878f-4ba23da92b07" containerID="a46e2670a81d69ebdba283cb2c37eeee84ef6189a48fb163e1741062d28000d3" exitCode=0 Nov 01 20:26:39.132092 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.131903 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" event={"ID":"1eb6f177-e8fa-43ab-878f-4ba23da92b07","Type":"ContainerDied","Data":"a46e2670a81d69ebdba283cb2c37eeee84ef6189a48fb163e1741062d28000d3"} Nov 01 20:26:39.504300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.504263 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l"] Nov 01 20:26:39.508730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.508707 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.566477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.566444 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l"] Nov 01 20:26:39.652611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.652576 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.652807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.652650 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlgrp\" (UniqueName: \"kubernetes.io/projected/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-kube-api-access-jlgrp\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.652807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.652788 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.754140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.754106 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.754401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.754180 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlgrp\" (UniqueName: \"kubernetes.io/projected/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-kube-api-access-jlgrp\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.754401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.754222 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.754951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.754924 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.756788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.756769 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.811170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.811136 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlgrp\" (UniqueName: \"kubernetes.io/projected/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-kube-api-access-jlgrp\") pod \"maas-api-revoke-sub-737debe1-44m6l\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:39.822894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:39.822860 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:40.219877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.219841 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l"] Nov 01 20:26:40.235188 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:26:40.235145 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0fdec9_77cd_49d0_ad3e_8b3898705b0b.slice/crio-2c7ee02a7a9394f08445eb2b54929e1c9d6e6fab497b42d086e54e763832e680 WatchSource:0}: Error finding container 2c7ee02a7a9394f08445eb2b54929e1c9d6e6fab497b42d086e54e763832e680: Status 404 returned error can't find the container with id 2c7ee02a7a9394f08445eb2b54929e1c9d6e6fab497b42d086e54e763832e680 Nov 01 20:26:40.294833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.294791 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:40.362640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.362607 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb6f177-e8fa-43ab-878f-4ba23da92b07-kube-api-access-h5s84\" (UniqueName: \"kubernetes.io/projected/1eb6f177-e8fa-43ab-878f-4ba23da92b07-kube-api-access-h5s84\") pod \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " Nov 01 20:26:40.362830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.362734 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-openshift-ca\") pod \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " Nov 01 20:26:40.362830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.362780 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-xks-ca\") pod \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\" (UID: \"1eb6f177-e8fa-43ab-878f-4ba23da92b07\") " Nov 01 20:26:40.363136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.363111 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-openshift-ca" pod "1eb6f177-e8fa-43ab-878f-4ba23da92b07" (UID: "1eb6f177-e8fa-43ab-878f-4ba23da92b07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:40.363271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.363256 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:40.364970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.364945 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-xks-ca" pod "1eb6f177-e8fa-43ab-878f-4ba23da92b07" (UID: "1eb6f177-e8fa-43ab-878f-4ba23da92b07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:40.365071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.364988 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb6f177-e8fa-43ab-878f-4ba23da92b07-kube-api-access-h5s84" pod "1eb6f177-e8fa-43ab-878f-4ba23da92b07" (UID: "1eb6f177-e8fa-43ab-878f-4ba23da92b07"). InnerVolumeSpecName "kube-api-access-h5s84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:40.467779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.467738 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5s84\" (UniqueName: \"kubernetes.io/projected/1eb6f177-e8fa-43ab-878f-4ba23da92b07-kube-api-access-h5s84\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:40.467958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:40.467788 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb6f177-e8fa-43ab-878f-4ba23da92b07-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:41.149599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:41.149558 5633 generic.go:356] "Generic (PLEG): container finished" podID="5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" containerID="ec9cc216bd1c34718e073f7641afd9f4ff3ef9b42db7244dfcca54a63e68e196" exitCode=0 Nov 01 20:26:41.150046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:41.149647 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" event={"ID":"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b","Type":"ContainerDied","Data":"ec9cc216bd1c34718e073f7641afd9f4ff3ef9b42db7244dfcca54a63e68e196"} Nov 01 20:26:41.150046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:41.149701 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" event={"ID":"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b","Type":"ContainerStarted","Data":"2c7ee02a7a9394f08445eb2b54929e1c9d6e6fab497b42d086e54e763832e680"} Nov 01 20:26:41.151159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:41.151138 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" Nov 01 20:26:41.151282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:41.151153 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4" event={"ID":"1eb6f177-e8fa-43ab-878f-4ba23da92b07","Type":"ContainerDied","Data":"6d7231da7137455cf21d03dcc4cb8853681884392917b1d50332e7cab07222f1"} Nov 01 20:26:41.151282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:41.151194 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d7231da7137455cf21d03dcc4cb8853681884392917b1d50332e7cab07222f1" Nov 01 20:26:41.282581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:41.282539 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:42.324436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.324398 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:42.494228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.494196 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-kube-api-access-jlgrp\" (UniqueName: \"kubernetes.io/projected/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-kube-api-access-jlgrp\") pod \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " Nov 01 20:26:42.494418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.494406 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-openshift-ca\") pod \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " Nov 01 20:26:42.494498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.494444 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-xks-ca\") pod \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\" (UID: \"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b\") " Nov 01 20:26:42.494796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.494772 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-openshift-ca" pod "5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" (UID: "5f0fdec9-77cd-49d0-ad3e-8b3898705b0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:42.494954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.494938 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:42.496485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.496459 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-kube-api-access-jlgrp" pod "5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" (UID: "5f0fdec9-77cd-49d0-ad3e-8b3898705b0b"). InnerVolumeSpecName "kube-api-access-jlgrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:42.496697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.496673 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-xks-ca" pod "5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" (UID: "5f0fdec9-77cd-49d0-ad3e-8b3898705b0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:42.596256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.596220 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlgrp\" (UniqueName: \"kubernetes.io/projected/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-kube-api-access-jlgrp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:42.596256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:42.596259 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:43.167655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:43.167624 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" Nov 01 20:26:43.167827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:43.167660 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l" event={"ID":"5f0fdec9-77cd-49d0-ad3e-8b3898705b0b","Type":"ContainerDied","Data":"2c7ee02a7a9394f08445eb2b54929e1c9d6e6fab497b42d086e54e763832e680"} Nov 01 20:26:43.167827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:43.167707 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c7ee02a7a9394f08445eb2b54929e1c9d6e6fab497b42d086e54e763832e680" Nov 01 20:26:45.999945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:45.999898 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:46.158460 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:26:46.158425 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:26:50.244527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.244495 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp"] Nov 01 20:26:50.245120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.245106 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" containerName="revoke-keys" Nov 01 20:26:50.245167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.245121 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" containerName="revoke-keys" Nov 01 20:26:50.245167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.245157 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1eb6f177-e8fa-43ab-878f-4ba23da92b07" containerName="revoke-keys" Nov 01 20:26:50.245167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.245162 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb6f177-e8fa-43ab-878f-4ba23da92b07" containerName="revoke-keys" Nov 01 20:26:50.245264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.245250 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb6f177-e8fa-43ab-878f-4ba23da92b07" containerName="revoke-keys" Nov 01 20:26:50.245264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.245262 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" containerName="revoke-keys" Nov 01 20:26:50.249262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.249244 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.299099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.299067 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.299285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.299242 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.299455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.299430 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9j9t\" (UniqueName: \"kubernetes.io/projected/51bfff28-88c0-4407-9581-c17ff9494ae3-kube-api-access-n9j9t\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.305436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.305393 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp"] Nov 01 20:26:50.348457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.348417 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:26:50.348457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.348425 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:26:50.348457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.348428 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:26:50.350577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.350556 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:26:50.401106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.401068 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.401290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.401158 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.401290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.401217 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9j9t\" (UniqueName: \"kubernetes.io/projected/51bfff28-88c0-4407-9581-c17ff9494ae3-kube-api-access-n9j9t\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.401930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.401876 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.403692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.403668 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.458261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.458223 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9j9t\" (UniqueName: \"kubernetes.io/projected/51bfff28-88c0-4407-9581-c17ff9494ae3-kube-api-access-n9j9t\") pod \"maas-api-revoke-sub-2b5d3d22-ckhjp\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.557907 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.557822 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:50.563701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.563677 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:50.696002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.695791 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:50.817806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.817701 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:50.972053 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:26:50.972021 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bfff28_88c0_4407_9581_c17ff9494ae3.slice/crio-ca58424d35cf865c27be0ffd3fa21c9735c306d601248bf5a019e7c7be44cc49 WatchSource:0}: Error finding container ca58424d35cf865c27be0ffd3fa21c9735c306d601248bf5a019e7c7be44cc49: Status 404 returned error can't find the container with id ca58424d35cf865c27be0ffd3fa21c9735c306d601248bf5a019e7c7be44cc49 Nov 01 20:26:50.972248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:50.972225 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp"] Nov 01 20:26:51.229693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:51.229659 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" event={"ID":"51bfff28-88c0-4407-9581-c17ff9494ae3","Type":"ContainerStarted","Data":"ca58424d35cf865c27be0ffd3fa21c9735c306d601248bf5a019e7c7be44cc49"} Nov 01 20:26:52.047557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:52.047527 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z"] Nov 01 20:26:52.096448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:52.096418 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0cc270-d5c9z"] Nov 01 20:26:52.239499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:52.239466 5633 generic.go:356] "Generic (PLEG): container finished" podID="51bfff28-88c0-4407-9581-c17ff9494ae3" containerID="c0ab39783d0c3229f720394bdb94d8700235c3ed5b05734eee979fe6e018b4bb" exitCode=0 Nov 01 20:26:52.239678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:52.239556 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" event={"ID":"51bfff28-88c0-4407-9581-c17ff9494ae3","Type":"ContainerDied","Data":"c0ab39783d0c3229f720394bdb94d8700235c3ed5b05734eee979fe6e018b4bb"} Nov 01 20:26:52.468846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:52.468815 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394f76af-9fca-496a-90c1-c49051bac02a" path="/var/lib/kubelet/pods/394f76af-9fca-496a-90c1-c49051bac02a/volumes" Nov 01 20:26:53.447026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.446996 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:53.559280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.559178 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51bfff28-88c0-4407-9581-c17ff9494ae3-kube-api-access-n9j9t\" (UniqueName: \"kubernetes.io/projected/51bfff28-88c0-4407-9581-c17ff9494ae3-kube-api-access-n9j9t\") pod \"51bfff28-88c0-4407-9581-c17ff9494ae3\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " Nov 01 20:26:53.559280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.559276 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-openshift-ca\") pod \"51bfff28-88c0-4407-9581-c17ff9494ae3\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " Nov 01 20:26:53.559557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.559324 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-xks-ca\") pod \"51bfff28-88c0-4407-9581-c17ff9494ae3\" (UID: \"51bfff28-88c0-4407-9581-c17ff9494ae3\") " Nov 01 20:26:53.560868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.560839 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-openshift-ca" pod "51bfff28-88c0-4407-9581-c17ff9494ae3" (UID: "51bfff28-88c0-4407-9581-c17ff9494ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:53.563601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.563560 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-xks-ca" pod "51bfff28-88c0-4407-9581-c17ff9494ae3" (UID: "51bfff28-88c0-4407-9581-c17ff9494ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:53.563601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.563569 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bfff28-88c0-4407-9581-c17ff9494ae3-kube-api-access-n9j9t" pod "51bfff28-88c0-4407-9581-c17ff9494ae3" (UID: "51bfff28-88c0-4407-9581-c17ff9494ae3"). InnerVolumeSpecName "kube-api-access-n9j9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:53.661717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.661681 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9j9t\" (UniqueName: \"kubernetes.io/projected/51bfff28-88c0-4407-9581-c17ff9494ae3-kube-api-access-n9j9t\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:53.661717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.661710 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:53.661717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:53.661722 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51bfff28-88c0-4407-9581-c17ff9494ae3-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:26:54.221731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:54.221695 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:26:54.266899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:54.266870 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" Nov 01 20:26:54.267101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:54.266902 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp" event={"ID":"51bfff28-88c0-4407-9581-c17ff9494ae3","Type":"ContainerDied","Data":"ca58424d35cf865c27be0ffd3fa21c9735c306d601248bf5a019e7c7be44cc49"} Nov 01 20:26:54.267101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:54.266978 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca58424d35cf865c27be0ffd3fa21c9735c306d601248bf5a019e7c7be44cc49" Nov 01 20:26:55.428146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:55.428048 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:56.093734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:56.093698 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd"] Nov 01 20:26:56.146355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:56.145894 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5732ec0-5zlkd"] Nov 01 20:26:56.513358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:56.493549 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a76671-86e0-4a7e-afa6-6d0649efceb2" path="/var/lib/kubelet/pods/67a76671-86e0-4a7e-afa6-6d0649efceb2/volumes" Nov 01 20:26:57.091362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:57.078347 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:26:59.863227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:26:59.863187 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:00.500342 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:00.500296 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:04.090777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:04.090734 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8"] Nov 01 20:27:04.139875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:04.139835 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b243f7-kg8q8"] Nov 01 20:27:04.468388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:04.468354 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d840bb0-fbb2-4c28-922f-b4ae8f82e008" path="/var/lib/kubelet/pods/4d840bb0-fbb2-4c28-922f-b4ae8f82e008/volumes" Nov 01 20:27:05.322114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:05.322078 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:08.058648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.058614 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9"] Nov 01 20:27:08.059272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.059255 5633 cpu_manager.go:414] "containerMap: removing container" podUID="51bfff28-88c0-4407-9581-c17ff9494ae3" containerName="revoke-keys" Nov 01 20:27:08.059366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.059274 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51bfff28-88c0-4407-9581-c17ff9494ae3" containerName="revoke-keys" Nov 01 20:27:08.059410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.059386 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="51bfff28-88c0-4407-9581-c17ff9494ae3" containerName="revoke-keys" Nov 01 20:27:08.063380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.063324 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.119243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.119210 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9"] Nov 01 20:27:08.147643 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.147596 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.147864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.147847 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdgs\" (UniqueName: \"kubernetes.io/projected/692a0ac6-6638-4b5f-b6ed-501ee613e59c-kube-api-access-dtdgs\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.148042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.148025 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.163309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.163269 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:27:08.163309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.163266 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:27:08.163309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.163302 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:27:08.163570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.163408 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:27:08.249614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.249575 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.249796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.249634 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.249850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.249806 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdgs\" (UniqueName: \"kubernetes.io/projected/692a0ac6-6638-4b5f-b6ed-501ee613e59c-kube-api-access-dtdgs\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.250394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.250367 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.252136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.252111 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.306371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.306324 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdgs\" (UniqueName: \"kubernetes.io/projected/692a0ac6-6638-4b5f-b6ed-501ee613e59c-kube-api-access-dtdgs\") pod \"maas-api-revoke-sub-7564a258-z94j9\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.377676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.377589 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:08.758209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:08.758177 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9"] Nov 01 20:27:08.758957 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:27:08.758931 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692a0ac6_6638_4b5f_b6ed_501ee613e59c.slice/crio-805570bc2ee1f9c0db493c4244b5d0ba4ccf2b0f1d7fe44c40e77d1c5f212c19 WatchSource:0}: Error finding container 805570bc2ee1f9c0db493c4244b5d0ba4ccf2b0f1d7fe44c40e77d1c5f212c19: Status 404 returned error can't find the container with id 805570bc2ee1f9c0db493c4244b5d0ba4ccf2b0f1d7fe44c40e77d1c5f212c19 Nov 01 20:27:09.485786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:09.485751 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" event={"ID":"692a0ac6-6638-4b5f-b6ed-501ee613e59c","Type":"ContainerStarted","Data":"255f1303c3237d22472abf07b7a3386e3db51a3b1f2820b9cc2bdd296254df29"} Nov 01 20:27:09.486192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:09.485811 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" event={"ID":"692a0ac6-6638-4b5f-b6ed-501ee613e59c","Type":"ContainerStarted","Data":"805570bc2ee1f9c0db493c4244b5d0ba4ccf2b0f1d7fe44c40e77d1c5f212c19"} Nov 01 20:27:09.600716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:09.600655 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" podStartSLOduration=1.600639745 podStartE2EDuration="1.600639745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:27:09.596765594 +0000 UTC m=+1437.271024843" watchObservedRunningTime="2026-11-01 20:27:09.600639745 +0000 UTC m=+1437.274899000" Nov 01 20:27:09.927732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:09.927692 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:10.495665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:10.495630 5633 generic.go:356] "Generic (PLEG): container finished" podID="692a0ac6-6638-4b5f-b6ed-501ee613e59c" containerID="255f1303c3237d22472abf07b7a3386e3db51a3b1f2820b9cc2bdd296254df29" exitCode=0 Nov 01 20:27:10.496104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:10.495721 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" event={"ID":"692a0ac6-6638-4b5f-b6ed-501ee613e59c","Type":"ContainerDied","Data":"255f1303c3237d22472abf07b7a3386e3db51a3b1f2820b9cc2bdd296254df29"} Nov 01 20:27:11.686006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.685978 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:11.798844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.798756 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-openshift-ca\") pod \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " Nov 01 20:27:11.798844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.798794 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692a0ac6-6638-4b5f-b6ed-501ee613e59c-kube-api-access-dtdgs\" (UniqueName: \"kubernetes.io/projected/692a0ac6-6638-4b5f-b6ed-501ee613e59c-kube-api-access-dtdgs\") pod \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " Nov 01 20:27:11.799040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.798933 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-xks-ca\") pod \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\" (UID: \"692a0ac6-6638-4b5f-b6ed-501ee613e59c\") " Nov 01 20:27:11.799188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.799162 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-openshift-ca" pod "692a0ac6-6638-4b5f-b6ed-501ee613e59c" (UID: "692a0ac6-6638-4b5f-b6ed-501ee613e59c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:27:11.799514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.799487 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:11.801085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.801061 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-xks-ca" pod "692a0ac6-6638-4b5f-b6ed-501ee613e59c" (UID: "692a0ac6-6638-4b5f-b6ed-501ee613e59c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:27:11.801192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.801127 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692a0ac6-6638-4b5f-b6ed-501ee613e59c-kube-api-access-dtdgs" pod "692a0ac6-6638-4b5f-b6ed-501ee613e59c" (UID: "692a0ac6-6638-4b5f-b6ed-501ee613e59c"). InnerVolumeSpecName "kube-api-access-dtdgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:27:11.900796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.900755 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtdgs\" (UniqueName: \"kubernetes.io/projected/692a0ac6-6638-4b5f-b6ed-501ee613e59c-kube-api-access-dtdgs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:11.900796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:11.900789 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692a0ac6-6638-4b5f-b6ed-501ee613e59c-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:12.512888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:12.512858 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" Nov 01 20:27:12.513071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:12.512890 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9" event={"ID":"692a0ac6-6638-4b5f-b6ed-501ee613e59c","Type":"ContainerDied","Data":"805570bc2ee1f9c0db493c4244b5d0ba4ccf2b0f1d7fe44c40e77d1c5f212c19"} Nov 01 20:27:12.513071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:12.512947 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805570bc2ee1f9c0db493c4244b5d0ba4ccf2b0f1d7fe44c40e77d1c5f212c19" Nov 01 20:27:14.812994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:14.812956 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:15.647358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:15.639914 5633 scope.go:122] "RemoveContainer" containerID="309046e23c4cd528ffd784148e5e32cc284ae1503bc30ffd29504028fb3435e3" Nov 01 20:27:15.733112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:15.733085 5633 scope.go:122] "RemoveContainer" containerID="404e790039987d781aa7e3ad31377b9588e236aad0d1df52aa853a2e7276f3ef" Nov 01 20:27:16.158877 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:27:16.158771 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:27:19.048445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:19.048411 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5"] Nov 01 20:27:19.098207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:19.098166 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cccee6-7ztt5"] Nov 01 20:27:19.706295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:19.706262 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:20.471380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:20.471310 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7761247-0c6c-43f3-aaa2-ecd182349257" path="/var/lib/kubelet/pods/d7761247-0c6c-43f3-aaa2-ecd182349257/volumes" Nov 01 20:27:22.415454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.415361 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:22.701458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.701378 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2"] Nov 01 20:27:22.702020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.701990 5633 cpu_manager.go:414] "containerMap: removing container" podUID="692a0ac6-6638-4b5f-b6ed-501ee613e59c" containerName="revoke-keys" Nov 01 20:27:22.702020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.702011 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692a0ac6-6638-4b5f-b6ed-501ee613e59c" containerName="revoke-keys" Nov 01 20:27:22.702176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.702134 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="692a0ac6-6638-4b5f-b6ed-501ee613e59c" containerName="revoke-keys" Nov 01 20:27:22.707402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.707380 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.753997 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-credential-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754056 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-podinfo\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754203 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-certs\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754243 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754374 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-token\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754418 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istiod-ca-cert\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754440 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-envoy\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754557 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ksh\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-kube-api-access-g2ksh\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.754611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.754599 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-data\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.759270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.759248 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-46c731c3-gw-openshift-default-dockercfg-xgptg\"" Nov 01 20:27:22.816475 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.816429 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2"] Nov 01 20:27:22.855653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.855614 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ksh\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-kube-api-access-g2ksh\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.855862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.855669 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-data\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.855862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.855750 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-credential-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.855862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.855800 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-podinfo\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.855862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.855852 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-certs\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856113 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856041 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856128 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-token\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856173 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-data\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856184 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istiod-ca-cert\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856187 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-certs\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856487 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856489 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-envoy\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.856946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856912 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-credential-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.857087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.856919 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istiod-ca-cert\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.858986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.858962 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-envoy\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.859566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.859547 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-podinfo\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.915897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.915865 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-token\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:22.919028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:22.918998 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ksh\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-kube-api-access-g2ksh\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:23.022677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.022645 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:23.453469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.453434 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2"] Nov 01 20:27:23.454068 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:27:23.454044 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d08b4a_aa10_4d35_843e_f5ec265a1f3e.slice/crio-b8153d10523e666f955730567b6a0087c5f768b2004614bd7c63f08fd540fd60 WatchSource:0}: Error finding container b8153d10523e666f955730567b6a0087c5f768b2004614bd7c63f08fd540fd60: Status 404 returned error can't find the container with id b8153d10523e666f955730567b6a0087c5f768b2004614bd7c63f08fd540fd60 Nov 01 20:27:23.456729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.456697 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:27:23.456803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.456784 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:27:23.456841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.456814 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:27:23.504285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.504257 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p"] Nov 01 20:27:23.509765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.509741 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.561673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.561640 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:27:23.561673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.561660 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:27:23.563845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.563816 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p"] Nov 01 20:27:23.568370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.565694 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.568370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.566029 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27rz\" (UniqueName: \"kubernetes.io/projected/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-kube-api-access-p27rz\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.568370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.566208 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.607557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.607516 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:27:23.607753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.607627 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:27:23.667416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.667383 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.667620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.667492 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.667692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.667640 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27rz\" (UniqueName: \"kubernetes.io/projected/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-kube-api-access-p27rz\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.668196 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.668166 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.670280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.670261 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.723693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.723655 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27rz\" (UniqueName: \"kubernetes.io/projected/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-kube-api-access-p27rz\") pod \"maas-api-revoke-sub-4717a66f-f698p\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.825529 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.825492 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:23.838121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.838075 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" event={"ID":"49d08b4a-aa10-4d35-843e-f5ec265a1f3e","Type":"ContainerStarted","Data":"ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21"} Nov 01 20:27:23.838284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.838141 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" event={"ID":"49d08b4a-aa10-4d35-843e-f5ec265a1f3e","Type":"ContainerStarted","Data":"b8153d10523e666f955730567b6a0087c5f768b2004614bd7c63f08fd540fd60"} Nov 01 20:27:23.955863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:23.955802 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podStartSLOduration=1.95578582 podStartE2EDuration="1.95578582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:27:23.955429956 +0000 UTC m=+1451.629689214" watchObservedRunningTime="2026-11-01 20:27:23.95578582 +0000 UTC m=+1451.630045140" Nov 01 20:27:24.025646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.024675 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:24.028611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.028576 5633 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 01 20:27:24.028782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.028649 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 01 20:27:24.222549 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:27:24.222511 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c4c57d_4d34_4ac8_9194_a56e2cc5ccab.slice/crio-7fe96977473647ea1db00113fea19525ca23d68c5385235d5f2cd75be55cbf4b WatchSource:0}: Error finding container 7fe96977473647ea1db00113fea19525ca23d68c5385235d5f2cd75be55cbf4b: Status 404 returned error can't find the container with id 7fe96977473647ea1db00113fea19525ca23d68c5385235d5f2cd75be55cbf4b Nov 01 20:27:24.222731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.222707 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p"] Nov 01 20:27:24.511866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.511825 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:24.860034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.859975 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" event={"ID":"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab","Type":"ContainerStarted","Data":"57039fa4d324b805ffe8aab43c5c7b2eaa6c2e0d3935baf9b650459a3d5f37a8"} Nov 01 20:27:24.860255 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.860131 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" event={"ID":"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab","Type":"ContainerStarted","Data":"7fe96977473647ea1db00113fea19525ca23d68c5385235d5f2cd75be55cbf4b"} Nov 01 20:27:24.974304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:24.974196 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" podStartSLOduration=1.974182222 podStartE2EDuration="1.974182222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:27:24.973098114 +0000 UTC m=+1452.647357370" watchObservedRunningTime="2026-11-01 20:27:24.974182222 +0000 UTC m=+1452.648441516" Nov 01 20:27:25.023647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:25.023613 5633 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 01 20:27:25.023839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:25.023668 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 01 20:27:25.612064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:25.611978 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:25.873437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:25.873321 5633 generic.go:356] "Generic (PLEG): container finished" podID="54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" containerID="57039fa4d324b805ffe8aab43c5c7b2eaa6c2e0d3935baf9b650459a3d5f37a8" exitCode=0 Nov 01 20:27:25.873437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:25.873358 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" event={"ID":"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab","Type":"ContainerDied","Data":"57039fa4d324b805ffe8aab43c5c7b2eaa6c2e0d3935baf9b650459a3d5f37a8"} Nov 01 20:27:26.022921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:26.022883 5633 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 01 20:27:26.023096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:26.022941 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 01 20:27:27.024980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.024944 5633 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 01 20:27:27.025383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.025026 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 01 20:27:27.126767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.126739 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:27.243263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.243228 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-kube-api-access-p27rz\" (UniqueName: \"kubernetes.io/projected/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-kube-api-access-p27rz\") pod \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " Nov 01 20:27:27.243478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.243294 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-xks-ca\") pod \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " Nov 01 20:27:27.243478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.243373 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-openshift-ca\") pod \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\" (UID: \"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab\") " Nov 01 20:27:27.244164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.244132 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-openshift-ca" pod "54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" (UID: "54c4c57d-4d34-4ac8-9194-a56e2cc5ccab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:27:27.246388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.246303 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-xks-ca" pod "54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" (UID: "54c4c57d-4d34-4ac8-9194-a56e2cc5ccab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:27:27.249989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.249956 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-kube-api-access-p27rz" pod "54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" (UID: "54c4c57d-4d34-4ac8-9194-a56e2cc5ccab"). InnerVolumeSpecName "kube-api-access-p27rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:27:27.345568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.345533 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p27rz\" (UniqueName: \"kubernetes.io/projected/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-kube-api-access-p27rz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:27.345568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.345562 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:27.345568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.345573 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:27.915551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.915524 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" Nov 01 20:27:27.915729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.915518 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p" event={"ID":"54c4c57d-4d34-4ac8-9194-a56e2cc5ccab","Type":"ContainerDied","Data":"7fe96977473647ea1db00113fea19525ca23d68c5385235d5f2cd75be55cbf4b"} Nov 01 20:27:27.915729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:27.915656 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe96977473647ea1db00113fea19525ca23d68c5385235d5f2cd75be55cbf4b" Nov 01 20:27:28.023512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:28.023476 5633 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 01 20:27:28.023688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:28.023536 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 01 20:27:29.023270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:29.023229 5633 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 01 20:27:29.023719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:29.023286 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 01 20:27:29.214551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:29.214514 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:30.028419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:30.028378 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:30.028825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:30.028804 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:30.029699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:30.029682 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:27:32.048568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:32.048535 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5"] Nov 01 20:27:32.097667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:32.097631 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-480dc2c4-bv6h5"] Nov 01 20:27:32.467747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:32.467709 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3a31ae-dca4-4000-99c7-ca1a1cb997aa" path="/var/lib/kubelet/pods/ba3a31ae-dca4-4000-99c7-ca1a1cb997aa/volumes" Nov 01 20:27:32.945704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:32.945613 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:27:33.910465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:33.910425 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:34.698429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:34.698389 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:35.331657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.331564 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:35.407395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.400632 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8"] Nov 01 20:27:35.407395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.401636 5633 cpu_manager.go:414] "containerMap: removing container" podUID="54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" containerName="revoke-keys" Nov 01 20:27:35.407395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.401659 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" containerName="revoke-keys" Nov 01 20:27:35.407395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.401888 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" containerName="revoke-keys" Nov 01 20:27:35.633907 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.633860 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8"] Nov 01 20:27:35.635126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.635100 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.750746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.750710 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-202d23-openshift-default-dockercfg-grkp8\"" Nov 01 20:27:35.791103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.791068 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.800849 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-token\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801036 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-socket\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801132 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-podinfo\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801159 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istiod-ca-cert\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801206 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-envoy\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801250 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tl6m\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-kube-api-access-8tl6m\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801397 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-credential-socket\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801543 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-data\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.809391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.801569 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-certs\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.905313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.905202 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-data\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.905584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.905560 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-certs\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.905768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.905752 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-token\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.905981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.905934 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-data\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.906161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906142 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-socket\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.906383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906363 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-podinfo\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.906521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906503 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istiod-ca-cert\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.906660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906644 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-envoy\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.906788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906772 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tl6m\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-kube-api-access-8tl6m\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.906971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906955 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-credential-socket\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.907497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.907469 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istiod-ca-cert\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.916451 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906819 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-socket\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.916853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.906219 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-certs\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.924746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.907477 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-credential-socket\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.924950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.924659 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-envoy\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.927665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.927601 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-podinfo\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:35.983060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:35.983027 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-token\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:36.002535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:36.002486 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tl6m\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-kube-api-access-8tl6m\") pod \"e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:36.085034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:36.084950 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:36.923276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:36.923193 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8"] Nov 01 20:27:36.963492 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:27:36.963459 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f2ae96_3051_4987_84f5_1ee9c3a1b48f.slice/crio-9ec6aa2042b697d357e2fcd4c0ae07df23bed9d40bda526c90d8b340adade774 WatchSource:0}: Error finding container 9ec6aa2042b697d357e2fcd4c0ae07df23bed9d40bda526c90d8b340adade774: Status 404 returned error can't find the container with id 9ec6aa2042b697d357e2fcd4c0ae07df23bed9d40bda526c90d8b340adade774 Nov 01 20:27:36.966147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:36.966111 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:27:36.966273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:36.966195 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:27:36.966273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:36.966242 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:27:37.110025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:37.109976 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" event={"ID":"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f","Type":"ContainerStarted","Data":"d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b"} Nov 01 20:27:37.110215 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:37.110046 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" event={"ID":"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f","Type":"ContainerStarted","Data":"9ec6aa2042b697d357e2fcd4c0ae07df23bed9d40bda526c90d8b340adade774"} Nov 01 20:27:37.227643 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:37.227590 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podStartSLOduration=2.227573364 podStartE2EDuration="2.227573364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:27:37.225011027 +0000 UTC m=+1464.899270283" watchObservedRunningTime="2026-11-01 20:27:37.227573364 +0000 UTC m=+1464.901832620" Nov 01 20:27:37.474732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:37.474688 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:38.085943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:38.085906 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:38.091968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:38.091935 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:38.118002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:38.117977 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:38.119107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:38.119087 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:27:38.714853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:38.714816 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:40.332036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:40.331996 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:40.541056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:40.541019 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:40.663698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:40.663614 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:43.061302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.061263 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk"] Nov 01 20:27:43.065752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.065734 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.088649 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.088618 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkc9t\" (UniqueName: \"kubernetes.io/projected/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-kube-api-access-qkc9t\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.088877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.088766 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.089487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.089458 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.170999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.170952 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:27:43.170999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.170983 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:27:43.170999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.170998 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:27:43.171374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.171171 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:27:43.176410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.176380 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk"] Nov 01 20:27:43.191565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.191508 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.192369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.192108 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkc9t\" (UniqueName: \"kubernetes.io/projected/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-kube-api-access-qkc9t\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.192369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.192213 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.193187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.193164 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.194849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.194827 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.248909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.248880 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkc9t\" (UniqueName: \"kubernetes.io/projected/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-kube-api-access-qkc9t\") pod \"maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.383182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.383102 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:43.760619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.760584 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk"] Nov 01 20:27:43.760767 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:27:43.760616 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381bd4dd_c78b_4ffb_bd6c_ed39c04a32b8.slice/crio-2cf6eeb0f4ec9a86d8b4332ac46de5aae97d0f3c2babd9441a5c34ceae6b3ec9 WatchSource:0}: Error finding container 2cf6eeb0f4ec9a86d8b4332ac46de5aae97d0f3c2babd9441a5c34ceae6b3ec9: Status 404 returned error can't find the container with id 2cf6eeb0f4ec9a86d8b4332ac46de5aae97d0f3c2babd9441a5c34ceae6b3ec9 Nov 01 20:27:43.831189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:43.831157 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:44.166230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:44.166190 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" event={"ID":"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8","Type":"ContainerStarted","Data":"2cf6eeb0f4ec9a86d8b4332ac46de5aae97d0f3c2babd9441a5c34ceae6b3ec9"} Nov 01 20:27:45.176613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:45.176525 5633 generic.go:356] "Generic (PLEG): container finished" podID="381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" containerID="6abb650fb707fc87d761ee7286d8151f128bd5156dd9a576ab1f02f9d3f89a64" exitCode=0 Nov 01 20:27:45.176613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:45.176568 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" event={"ID":"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8","Type":"ContainerDied","Data":"6abb650fb707fc87d761ee7286d8151f128bd5156dd9a576ab1f02f9d3f89a64"} Nov 01 20:27:45.630225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:45.630187 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:45.800526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:45.800490 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:46.264975 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:27:46.264941 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:27:46.426299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.426270 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:46.506783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.506699 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-kube-api-access-qkc9t\" (UniqueName: \"kubernetes.io/projected/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-kube-api-access-qkc9t\") pod \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " Nov 01 20:27:46.506783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.506758 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-xks-ca\") pod \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " Nov 01 20:27:46.507033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.506796 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-openshift-ca\") pod \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\" (UID: \"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8\") " Nov 01 20:27:46.507223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.507193 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-openshift-ca" pod "381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" (UID: "381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:27:46.509068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.509045 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-xks-ca" pod "381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" (UID: "381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:27:46.509147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.509074 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-kube-api-access-qkc9t" pod "381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" (UID: "381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8"). InnerVolumeSpecName "kube-api-access-qkc9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:27:46.609128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.609097 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkc9t\" (UniqueName: \"kubernetes.io/projected/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-kube-api-access-qkc9t\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:46.609128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.609121 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:46.609128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:46.609131 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:47.257587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:47.257547 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" event={"ID":"381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8","Type":"ContainerDied","Data":"2cf6eeb0f4ec9a86d8b4332ac46de5aae97d0f3c2babd9441a5c34ceae6b3ec9"} Nov 01 20:27:47.257587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:47.257548 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk" Nov 01 20:27:47.257871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:47.257615 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf6eeb0f4ec9a86d8b4332ac46de5aae97d0f3c2babd9441a5c34ceae6b3ec9" Nov 01 20:27:49.005827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:49.005787 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:51.100595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:51.089476 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:27:51.100595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:51.089838 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:27:54.044188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:54.042271 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:54.884011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:54.883974 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:55.008436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:55.008387 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:57.047390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:57.047354 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt"] Nov 01 20:27:57.096518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:57.096486 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f42b71e-trmkt"] Nov 01 20:27:58.467964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:58.467924 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93f0813-21c0-4426-8f75-ffdcdee9efdd" path="/var/lib/kubelet/pods/b93f0813-21c0-4426-8f75-ffdcdee9efdd/volumes" Nov 01 20:27:58.704728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:58.704689 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:27:59.868209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:27:59.868173 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:28:01.799654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:01.799613 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:03.390956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:03.390913 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:06.737614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.737580 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc"] Nov 01 20:28:06.738438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.738413 5633 cpu_manager.go:414] "containerMap: removing container" podUID="381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" containerName="revoke-keys" Nov 01 20:28:06.738438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.738436 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" containerName="revoke-keys" Nov 01 20:28:06.738649 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.738617 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" containerName="revoke-keys" Nov 01 20:28:06.742611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.742587 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.798375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.798323 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc"] Nov 01 20:28:06.841772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.841743 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:28:06.841772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.841742 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:28:06.842012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.841787 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:28:06.842012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.841787 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:28:06.846736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.846707 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.846845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.846748 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.846845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.846776 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4tjz\" (UniqueName: \"kubernetes.io/projected/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-kube-api-access-c4tjz\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.948308 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.948260 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.948566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.948323 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.948566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.948410 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4tjz\" (UniqueName: \"kubernetes.io/projected/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-kube-api-access-c4tjz\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.949085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.949064 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:06.951359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:06.951342 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:07.005324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:07.005233 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4tjz\" (UniqueName: \"kubernetes.io/projected/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-kube-api-access-c4tjz\") pod \"maas-api-revoke-keys-e2e-mt-71335882-sdkcc\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:07.056622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:07.056584 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:07.442914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:07.442813 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc"] Nov 01 20:28:07.450709 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:28:07.450677 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92cc3ce_a112_4347_b9d3_80e6b3c1ccc8.slice/crio-86e5127f9d21382d65191a5d9c67d8160a5ba65d808e9b9caba684332da83f11 WatchSource:0}: Error finding container 86e5127f9d21382d65191a5d9c67d8160a5ba65d808e9b9caba684332da83f11: Status 404 returned error can't find the container with id 86e5127f9d21382d65191a5d9c67d8160a5ba65d808e9b9caba684332da83f11 Nov 01 20:28:07.559250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:07.559204 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" event={"ID":"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8","Type":"ContainerStarted","Data":"86e5127f9d21382d65191a5d9c67d8160a5ba65d808e9b9caba684332da83f11"} Nov 01 20:28:08.328305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:08.328266 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:08.573527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:08.573435 5633 generic.go:356] "Generic (PLEG): container finished" podID="b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" containerID="66a066e32d2a8e36222ed22b4e3b7f7f447caa89a235cd1458b98a92f022b4bc" exitCode=0 Nov 01 20:28:08.573527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:08.573515 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" event={"ID":"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8","Type":"ContainerDied","Data":"66a066e32d2a8e36222ed22b4e3b7f7f447caa89a235cd1458b98a92f022b4bc"} Nov 01 20:28:09.464103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.464065 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:09.593400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.593364 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:09.904147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.904092 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:09.915803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.915767 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-kube-api-access-c4tjz\" (UniqueName: \"kubernetes.io/projected/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-kube-api-access-c4tjz\") pod \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " Nov 01 20:28:09.916017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.915826 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-openshift-ca\") pod \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " Nov 01 20:28:09.916017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.915971 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-xks-ca\") pod \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\" (UID: \"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8\") " Nov 01 20:28:09.916237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.916215 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-openshift-ca" pod "b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" (UID: "b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:09.916606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.916581 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:09.918435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.918399 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-kube-api-access-c4tjz" pod "b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" (UID: "b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8"). InnerVolumeSpecName "kube-api-access-c4tjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:09.919402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:09.919372 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-xks-ca" pod "b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" (UID: "b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:28:10.017711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:10.017679 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:10.017711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:10.017711 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4tjz\" (UniqueName: \"kubernetes.io/projected/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8-kube-api-access-c4tjz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:10.591708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:10.591277 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" event={"ID":"b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8","Type":"ContainerDied","Data":"86e5127f9d21382d65191a5d9c67d8160a5ba65d808e9b9caba684332da83f11"} Nov 01 20:28:10.591708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:10.591358 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e5127f9d21382d65191a5d9c67d8160a5ba65d808e9b9caba684332da83f11" Nov 01 20:28:10.591708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:10.591368 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc" Nov 01 20:28:13.569242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:13.569209 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:15.945474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:15.945432 5633 scope.go:122] "RemoveContainer" containerID="e0007837601a6589ac1176ea5810ccacec97493e22f4b58cc86ebf451171e7e0" Nov 01 20:28:15.960921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:15.960889 5633 scope.go:122] "RemoveContainer" containerID="b52665af1805b8b3470456a1f7e65a5a09a352ab9987445e036eb0e23c6d8cd6" Nov 01 20:28:15.976612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:15.976590 5633 scope.go:122] "RemoveContainer" containerID="86dbceaf602e188d1f6a8a1a3e7b543aebfb8cf757638a849897a93d74245ef1" Nov 01 20:28:15.987989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:15.987967 5633 scope.go:122] "RemoveContainer" containerID="a24b0eb11ff231831a75a1457fbd39a5022cdd000afc072e06923d049ca465cb" Nov 01 20:28:15.998938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:15.998916 5633 scope.go:122] "RemoveContainer" containerID="62bba7dd27829c3ebe18048ae6f13b3c4db153efac3f26495765eda6b4c0370a" Nov 01 20:28:15.999274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:15.999241 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:16.009951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:16.009929 5633 scope.go:122] "RemoveContainer" containerID="79dd18c8ec01da41073ecf1db0a2099d8f9ff3579e211bcdb4d318946da1c833" Nov 01 20:28:16.158438 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:28:16.158400 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:28:17.763134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:17.763097 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:17.886500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:17.886460 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:18.556282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:18.552024 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:18.678864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:18.678830 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:19.446604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:19.446572 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:19.655277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:19.655236 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:19.780933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:19.780900 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:19.906387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:19.906307 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:20.021050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:20.021017 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2"] Nov 01 20:28:20.021285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:20.021254 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" containerID="cri-o://ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21" gracePeriod=30 Nov 01 20:28:20.026216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:20.025927 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk"] Nov 01 20:28:20.030859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:20.030832 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46c731c3-rrpvk"] Nov 01 20:28:20.051630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:20.051531 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:20.476125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:20.476093 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8" path="/var/lib/kubelet/pods/381bd4dd-c78b-4ffb-bd6c-ed39c04a32b8/volumes" Nov 01 20:28:20.631844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:20.631806 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:21.261281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.260120 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6"] Nov 01 20:28:21.261281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.261065 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" containerName="revoke-keys" Nov 01 20:28:21.261281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.261086 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" containerName="revoke-keys" Nov 01 20:28:21.261281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.261279 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" containerName="revoke-keys" Nov 01 20:28:21.268858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.268818 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.309524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.309491 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6"] Nov 01 20:28:21.321689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.321650 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-podinfo\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.321910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.321722 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwhr\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-kube-api-access-2xwhr\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.321910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.321877 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-certs\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.322049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.321924 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-token\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.322049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.321968 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istiod-ca-cert\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.322160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.322049 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-socket\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.322160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.322099 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-envoy\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.322160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.322137 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-credential-socket\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.322420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.322200 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-data\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.336283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.336236 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:21.352453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.352418 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7387069c-openshift-default-dockercfg-pv7wc\"" Nov 01 20:28:21.423585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423541 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-certs\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.423793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423599 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-token\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.423793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423649 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istiod-ca-cert\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.423793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423720 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-socket\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.423793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423765 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-envoy\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.423793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423786 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-credential-socket\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423840 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-data\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423955 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-podinfo\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.423991 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-certs\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.424034 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwhr\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-kube-api-access-2xwhr\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.424282 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-socket\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.424438 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-credential-socket\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.424471 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-data\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.424680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.424662 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istiod-ca-cert\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.426429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.426411 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-envoy\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.426714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.426698 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-podinfo\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.475913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.475865 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:21.482678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.482650 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-token\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.483091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.483063 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwhr\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-kube-api-access-2xwhr\") pod \"e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:21.587584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:21.587489 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:22.022670 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:28:22.022635 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e9fdd8_9d3f_44be_8d60_bd816153cf6d.slice/crio-0ae2e85f8ad1de75c12785434a0fcb91d9599109eaf1da99b2d2bc2cd47a7b23 WatchSource:0}: Error finding container 0ae2e85f8ad1de75c12785434a0fcb91d9599109eaf1da99b2d2bc2cd47a7b23: Status 404 returned error can't find the container with id 0ae2e85f8ad1de75c12785434a0fcb91d9599109eaf1da99b2d2bc2cd47a7b23 Nov 01 20:28:22.023286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:22.023261 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6"] Nov 01 20:28:22.025183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:22.025153 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:28:22.025271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:22.025216 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:28:22.025271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:22.025246 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:28:22.710069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:22.710028 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" event={"ID":"60e9fdd8-9d3f-44be-8d60-bd816153cf6d","Type":"ContainerStarted","Data":"39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0"} Nov 01 20:28:22.710596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:22.710092 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" event={"ID":"60e9fdd8-9d3f-44be-8d60-bd816153cf6d","Type":"ContainerStarted","Data":"0ae2e85f8ad1de75c12785434a0fcb91d9599109eaf1da99b2d2bc2cd47a7b23"} Nov 01 20:28:22.926731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:22.926673 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podStartSLOduration=1.926654576 podStartE2EDuration="1.926654576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:22.875352061 +0000 UTC m=+1510.549611314" watchObservedRunningTime="2026-11-01 20:28:22.926654576 +0000 UTC m=+1510.600913832" Nov 01 20:28:23.405771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:23.405717 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:23.559864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:23.559828 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:23.600473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:23.595145 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:23.605746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:23.605718 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:23.733367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:23.732543 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:23.733848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:23.733698 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:28:24.382215 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:24.382178 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:28:25.114018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.113959 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:25.350100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.350073 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:28:25.382181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.382145 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt"] Nov 01 20:28:25.444231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.444192 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:25.492659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.492621 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-credential-socket\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.493018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.492736 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istiod-ca-cert\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.493018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.492788 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-envoy\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.493018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.492821 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-socket\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.493018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.492877 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-token\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-token\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494049 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-credential-socket" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494112 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-data\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494212 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-podinfo\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494213 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istiod-ca-cert" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494250 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-certs\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494385 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-kube-api-access-g2ksh\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-kube-api-access-g2ksh\") pod \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\" (UID: \"49d08b4a-aa10-4d35-843e-f5ec265a1f3e\") " Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494681 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-certs" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.494736 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-socket" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:25.495693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.495386 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-data" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:25.498709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.498683 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.498814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.498715 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.498814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.498730 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.498814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.498744 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.498814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.498758 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.499208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.499178 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-kube-api-access-g2ksh" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "kube-api-access-g2ksh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:25.506058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.506024 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-token" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:25.507830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.507798 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-envoy" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:25.507954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.507858 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-podinfo" pod "49d08b4a-aa10-4d35-843e-f5ec265a1f3e" (UID: "49d08b4a-aa10-4d35-843e-f5ec265a1f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:28:25.602008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.601927 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.602008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.601958 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2ksh\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-kube-api-access-g2ksh\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.602008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.601971 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.602008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.601983 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d08b4a-aa10-4d35-843e-f5ec265a1f3e-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:25.635854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.635802 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:25.749448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.749411 5633 generic.go:356] "Generic (PLEG): container finished" podID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerID="ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21" exitCode=0 Nov 01 20:28:25.749626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.749472 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" Nov 01 20:28:25.749626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.749493 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" event={"ID":"49d08b4a-aa10-4d35-843e-f5ec265a1f3e","Type":"ContainerDied","Data":"ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21"} Nov 01 20:28:25.749626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.749572 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2" event={"ID":"49d08b4a-aa10-4d35-843e-f5ec265a1f3e","Type":"ContainerDied","Data":"b8153d10523e666f955730567b6a0087c5f768b2004614bd7c63f08fd540fd60"} Nov 01 20:28:25.749626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.749603 5633 scope.go:122] "RemoveContainer" containerID="ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21" Nov 01 20:28:25.762598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.762575 5633 scope.go:122] "RemoveContainer" containerID="ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21" Nov 01 20:28:25.763034 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:28:25.763007 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21\": container with ID starting with ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21 not found: ID does not exist" containerID="ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21" Nov 01 20:28:25.763111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.763050 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21"} err="failed to get container status \"ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21\": rpc error: code = NotFound desc = could not find container \"ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21\": container with ID starting with ee724d9204c8f07fee91ccf0a49b0ec48a9d90fe33961b04edb27271ca58ce21 not found: ID does not exist" Nov 01 20:28:25.772393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.772362 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:25.919258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.919181 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2"] Nov 01 20:28:25.923871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.923845 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-bmpb2"] Nov 01 20:28:25.935264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:25.935231 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:26.468433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:26.468391 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" path="/var/lib/kubelet/pods/49d08b4a-aa10-4d35-843e-f5ec265a1f3e/volumes" Nov 01 20:28:27.123744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:27.123708 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:30.292864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.292826 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:30.383448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.383401 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" podUID="4dd39436-7442-42f3-aa2d-5abb664f2415" containerName="payload-processing" containerID="cri-o://c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f" gracePeriod=25 Nov 01 20:28:30.664513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.664487 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:28:30.773556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.773521 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-routing-overlay-volume\") pod \"4dd39436-7442-42f3-aa2d-5abb664f2415\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " Nov 01 20:28:30.773758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.773584 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-plugins-config-volume\") pod \"4dd39436-7442-42f3-aa2d-5abb664f2415\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " Nov 01 20:28:30.773758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.773668 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-provider-credentials\") pod \"4dd39436-7442-42f3-aa2d-5abb664f2415\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " Nov 01 20:28:30.773758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.773726 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-kube-api-access-6v9x9\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-kube-api-access-6v9x9\") pod \"4dd39436-7442-42f3-aa2d-5abb664f2415\" (UID: \"4dd39436-7442-42f3-aa2d-5abb664f2415\") " Nov 01 20:28:30.773954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.773927 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-routing-overlay-volume" pod "4dd39436-7442-42f3-aa2d-5abb664f2415" (UID: "4dd39436-7442-42f3-aa2d-5abb664f2415"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:30.774087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.774049 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-plugins-config-volume" pod "4dd39436-7442-42f3-aa2d-5abb664f2415" (UID: "4dd39436-7442-42f3-aa2d-5abb664f2415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:30.774310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.774288 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:30.774425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.774312 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd39436-7442-42f3-aa2d-5abb664f2415-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:30.776112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.776089 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-kube-api-access-6v9x9" pod "4dd39436-7442-42f3-aa2d-5abb664f2415" (UID: "4dd39436-7442-42f3-aa2d-5abb664f2415"). InnerVolumeSpecName "kube-api-access-6v9x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:30.776194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.776115 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-provider-credentials" pod "4dd39436-7442-42f3-aa2d-5abb664f2415" (UID: "4dd39436-7442-42f3-aa2d-5abb664f2415"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:30.802289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.802255 5633 generic.go:356] "Generic (PLEG): container finished" podID="4dd39436-7442-42f3-aa2d-5abb664f2415" containerID="c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f" exitCode=0 Nov 01 20:28:30.802499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.802324 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" Nov 01 20:28:30.802499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.802362 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" event={"ID":"4dd39436-7442-42f3-aa2d-5abb664f2415","Type":"ContainerDied","Data":"c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f"} Nov 01 20:28:30.802499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.802419 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt" event={"ID":"4dd39436-7442-42f3-aa2d-5abb664f2415","Type":"ContainerDied","Data":"451556235f0ad963ca98b31d5395e2785294b3132952f1c54772de9c4ad9b0af"} Nov 01 20:28:30.802499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.802446 5633 scope.go:122] "RemoveContainer" containerID="c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f" Nov 01 20:28:30.815147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.814826 5633 scope.go:122] "RemoveContainer" containerID="c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f" Nov 01 20:28:30.815274 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:28:30.815246 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f\": container with ID starting with c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f not found: ID does not exist" containerID="c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f" Nov 01 20:28:30.815393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.815289 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f"} err="failed to get container status \"c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f\": rpc error: code = NotFound desc = could not find container \"c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f\": container with ID starting with c5dca146e3d416a5960a0ede7a9b54b9895a057b973a7223d3b66c20a1f95b4f not found: ID does not exist" Nov 01 20:28:30.876044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.876011 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:30.876044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.876041 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9x9\" (UniqueName: \"kubernetes.io/projected/4dd39436-7442-42f3-aa2d-5abb664f2415-kube-api-access-6v9x9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:30.966912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.966867 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt"] Nov 01 20:28:30.972095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:30.972070 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d8p9qt"] Nov 01 20:28:32.468323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:32.468289 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd39436-7442-42f3-aa2d-5abb664f2415" path="/var/lib/kubelet/pods/4dd39436-7442-42f3-aa2d-5abb664f2415/volumes" Nov 01 20:28:33.146234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:33.146192 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:33.333595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:33.333557 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:33.469041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:33.469006 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:33.600256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:33.600209 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:34.559138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:34.559101 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:34.726023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:34.725988 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:35.478028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:35.477984 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:37.136241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:37.136211 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:37.264847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:37.264815 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:39.704379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.704315 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8"] Nov 01 20:28:39.705365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.705312 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4dd39436-7442-42f3-aa2d-5abb664f2415" containerName="payload-processing" Nov 01 20:28:39.705365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.705360 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd39436-7442-42f3-aa2d-5abb664f2415" containerName="payload-processing" Nov 01 20:28:39.705525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.705415 5633 cpu_manager.go:414] "containerMap: removing container" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" Nov 01 20:28:39.705525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.705423 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" Nov 01 20:28:39.705596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.705562 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d08b4a-aa10-4d35-843e-f5ec265a1f3e" containerName="istio-proxy" Nov 01 20:28:39.705596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.705584 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd39436-7442-42f3-aa2d-5abb664f2415" containerName="payload-processing" Nov 01 20:28:39.711627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.711599 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.755643 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.755602 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.755982 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.755962 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvqw\" (UniqueName: \"kubernetes.io/projected/9057688e-8632-407f-beab-acf487a0b1d2-kube-api-access-9tvqw\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.756241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.756223 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.761567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.761538 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8"] Nov 01 20:28:39.805843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.805811 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:28:39.806307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.806277 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:28:39.813477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.813440 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:28:39.813680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.813449 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:28:39.857971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.857935 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.858187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.858000 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvqw\" (UniqueName: \"kubernetes.io/projected/9057688e-8632-407f-beab-acf487a0b1d2-kube-api-access-9tvqw\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.858187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.858114 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.858966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.858936 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.860638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.860612 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:39.914849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:39.914806 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvqw\" (UniqueName: \"kubernetes.io/projected/9057688e-8632-407f-beab-acf487a0b1d2-kube-api-access-9tvqw\") pod \"maas-api-revoke-sub-a8184ce1-4dch8\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:40.029659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:40.029619 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:40.420876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:40.420842 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8"] Nov 01 20:28:40.425192 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:28:40.425157 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9057688e_8632_407f_beab_acf487a0b1d2.slice/crio-5affd09b48eea7042de02b180dfcab5463c6c8f41aa03f9f752c4347537bbcc5 WatchSource:0}: Error finding container 5affd09b48eea7042de02b180dfcab5463c6c8f41aa03f9f752c4347537bbcc5: Status 404 returned error can't find the container with id 5affd09b48eea7042de02b180dfcab5463c6c8f41aa03f9f752c4347537bbcc5 Nov 01 20:28:40.474756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:40.474722 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:41.030139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:41.030095 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" event={"ID":"9057688e-8632-407f-beab-acf487a0b1d2","Type":"ContainerStarted","Data":"5affd09b48eea7042de02b180dfcab5463c6c8f41aa03f9f752c4347537bbcc5"} Nov 01 20:28:42.040188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:42.040101 5633 generic.go:356] "Generic (PLEG): container finished" podID="9057688e-8632-407f-beab-acf487a0b1d2" containerID="294bc6c36303f65ab0a687240c0590cdf85c0f2c7ba0ac892ae3dc10f377d934" exitCode=0 Nov 01 20:28:42.040617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:42.040180 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" event={"ID":"9057688e-8632-407f-beab-acf487a0b1d2","Type":"ContainerDied","Data":"294bc6c36303f65ab0a687240c0590cdf85c0f2c7ba0ac892ae3dc10f377d934"} Nov 01 20:28:43.247884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.247856 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:43.332185 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.332089 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-xks-ca\") pod \"9057688e-8632-407f-beab-acf487a0b1d2\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " Nov 01 20:28:43.332396 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.332284 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-openshift-ca\") pod \"9057688e-8632-407f-beab-acf487a0b1d2\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " Nov 01 20:28:43.332502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.332454 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9057688e-8632-407f-beab-acf487a0b1d2-kube-api-access-9tvqw\" (UniqueName: \"kubernetes.io/projected/9057688e-8632-407f-beab-acf487a0b1d2-kube-api-access-9tvqw\") pod \"9057688e-8632-407f-beab-acf487a0b1d2\" (UID: \"9057688e-8632-407f-beab-acf487a0b1d2\") " Nov 01 20:28:43.332640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.332615 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-openshift-ca" pod "9057688e-8632-407f-beab-acf487a0b1d2" (UID: "9057688e-8632-407f-beab-acf487a0b1d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:43.332939 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.332920 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:43.334503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.334473 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-xks-ca" pod "9057688e-8632-407f-beab-acf487a0b1d2" (UID: "9057688e-8632-407f-beab-acf487a0b1d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:28:43.334779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.334759 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9057688e-8632-407f-beab-acf487a0b1d2-kube-api-access-9tvqw" pod "9057688e-8632-407f-beab-acf487a0b1d2" (UID: "9057688e-8632-407f-beab-acf487a0b1d2"). InnerVolumeSpecName "kube-api-access-9tvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:43.434287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.434248 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tvqw\" (UniqueName: \"kubernetes.io/projected/9057688e-8632-407f-beab-acf487a0b1d2-kube-api-access-9tvqw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:43.434287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:43.434283 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9057688e-8632-407f-beab-acf487a0b1d2-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:44.057806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:44.057767 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" event={"ID":"9057688e-8632-407f-beab-acf487a0b1d2","Type":"ContainerDied","Data":"5affd09b48eea7042de02b180dfcab5463c6c8f41aa03f9f752c4347537bbcc5"} Nov 01 20:28:44.057806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:44.057808 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8" Nov 01 20:28:44.058023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:44.057829 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5affd09b48eea7042de02b180dfcab5463c6c8f41aa03f9f752c4347537bbcc5" Nov 01 20:28:45.219505 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:45.219463 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:46.157425 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:28:46.157376 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:28:49.875381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:49.875324 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:50.134017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:50.133930 5633 generic.go:356] "Generic (PLEG): container finished" podID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerID="d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b" exitCode=0 Nov 01 20:28:50.134017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:50.134002 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" event={"ID":"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f","Type":"ContainerDied","Data":"d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b"} Nov 01 20:28:50.134477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:50.134462 5633 scope.go:122] "RemoveContainer" containerID="d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b" Nov 01 20:28:50.136058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:50.136030 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:28:50.136147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:50.136117 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:28:50.136189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:50.136149 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:28:50.970004 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:50.969966 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:28:51.147091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:51.147004 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" event={"ID":"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f","Type":"ContainerStarted","Data":"8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f"} Nov 01 20:28:52.090756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:52.090720 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:28:52.091545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:52.091512 5633 patch_prober.go:29] interesting pod/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:28:52.091788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:52.091737 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:28:53.052606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:53.052562 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc"] Nov 01 20:28:53.102358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:53.085378 5633 patch_prober.go:29] interesting pod/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:28:53.103598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:53.103549 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:28:53.118371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:53.118312 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:28:53.120691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:53.120657 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7"] Nov 01 20:28:53.124873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:53.124843 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38f9c29-dm7nc"] Nov 01 20:28:53.129063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:53.129028 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf6932e9-j4wk7"] Nov 01 20:28:54.094370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:54.093779 5633 patch_prober.go:29] interesting pod/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:28:54.094370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:54.093847 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:28:54.470710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:54.470673 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbbfa87-4335-404e-a86d-c83b8e9e5591" path="/var/lib/kubelet/pods/4cbbfa87-4335-404e-a86d-c83b8e9e5591/volumes" Nov 01 20:28:54.471382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:54.471325 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d064b2fd-3070-4712-9f0a-471b3d1e837c" path="/var/lib/kubelet/pods/d064b2fd-3070-4712-9f0a-471b3d1e837c/volumes" Nov 01 20:28:54.712931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:54.712892 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:28:55.095358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:55.089822 5633 patch_prober.go:29] interesting pod/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:28:55.095358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:55.089905 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:28:56.110543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:56.110503 5633 patch_prober.go:29] interesting pod/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 20:28:56.111030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:56.110574 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 20:28:57.090502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:57.090470 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:28:57.091781 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:57.091756 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:28:59.483566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:28:59.483518 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:04.130609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:04.130555 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:04.304978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:04.304899 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:04.440380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:04.440220 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:06.048271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:06.048236 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp"] Nov 01 20:29:06.097427 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:06.097389 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05c3623-x66sp"] Nov 01 20:29:06.262394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:06.262351 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:06.469786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:06.469751 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8550344c-79a8-45a3-96f4-a49ad4c2cd4c" path="/var/lib/kubelet/pods/8550344c-79a8-45a3-96f4-a49ad4c2cd4c/volumes" Nov 01 20:29:09.046659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:09.046627 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq"] Nov 01 20:29:09.095486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:09.095452 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c84570-z4kgq"] Nov 01 20:29:09.377609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:09.377514 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:10.468248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.468214 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97dd085-e0fb-476c-b164-4e30174c1352" path="/var/lib/kubelet/pods/c97dd085-e0fb-476c-b164-4e30174c1352/volumes" Nov 01 20:29:10.938594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.938496 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv"] Nov 01 20:29:10.939136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.939122 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9057688e-8632-407f-beab-acf487a0b1d2" containerName="revoke-keys" Nov 01 20:29:10.939136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.939136 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9057688e-8632-407f-beab-acf487a0b1d2" containerName="revoke-keys" Nov 01 20:29:10.939258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.939217 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9057688e-8632-407f-beab-acf487a0b1d2" containerName="revoke-keys" Nov 01 20:29:10.943115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.943098 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:10.951139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.951104 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:10.951900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.951874 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv"] Nov 01 20:29:10.995364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.995313 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-202d23\"/\"kube-root-ca.crt\"" Nov 01 20:29:10.995558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.995438 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-202d23\"/\"default-dockercfg-4sjc7\"" Nov 01 20:29:10.995558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.995486 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-202d23\"/\"openshift-service-ca.crt\"" Nov 01 20:29:10.995661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:10.995562 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-202d23\"/\"autores-model-202d23-kserve-self-signed-certs\"" Nov 01 20:29:11.063527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.063314 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-dshm\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.064544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.064494 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-home\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.064805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.064784 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd8961e-6222-49dd-adcc-d0925e90732a-tls-certs\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.064899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.064841 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-tmp-dir\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.065024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.064995 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4flj\" (UniqueName: \"kubernetes.io/projected/2bd8961e-6222-49dd-adcc-d0925e90732a-kube-api-access-m4flj\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.065117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.065068 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-model-cache\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.166584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.166548 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-home\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.166769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.166633 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd8961e-6222-49dd-adcc-d0925e90732a-tls-certs\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.166769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.166655 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-tmp-dir\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.166769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.166675 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4flj\" (UniqueName: \"kubernetes.io/projected/2bd8961e-6222-49dd-adcc-d0925e90732a-kube-api-access-m4flj\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.166769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.166690 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-model-cache\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.166996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.166799 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-dshm\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.167063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.167012 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-home\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.167130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.167069 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-tmp-dir\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.167199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.167152 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-model-cache\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.169180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.169160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-dshm\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.169383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.169362 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd8961e-6222-49dd-adcc-d0925e90732a-tls-certs\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.223115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.223078 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4flj\" (UniqueName: \"kubernetes.io/projected/2bd8961e-6222-49dd-adcc-d0925e90732a-kube-api-access-m4flj\") pod \"autores-model-202d23-kserve-85f8f55d45-twzqv\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.260140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.260105 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:11.691562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:11.691533 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv"] Nov 01 20:29:11.692113 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:29:11.692085 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bd8961e_6222_49dd_adcc_d0925e90732a.slice/crio-10dfa9fa10ab17bd5e104fd2dfbeaf1b71842ed08af6a4fadf94866d8f4d03f2 WatchSource:0}: Error finding container 10dfa9fa10ab17bd5e104fd2dfbeaf1b71842ed08af6a4fadf94866d8f4d03f2: Status 404 returned error can't find the container with id 10dfa9fa10ab17bd5e104fd2dfbeaf1b71842ed08af6a4fadf94866d8f4d03f2 Nov 01 20:29:12.407502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:12.407463 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" event={"ID":"2bd8961e-6222-49dd-adcc-d0925e90732a","Type":"ContainerStarted","Data":"01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e"} Nov 01 20:29:12.407708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:12.407510 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" event={"ID":"2bd8961e-6222-49dd-adcc-d0925e90732a","Type":"ContainerStarted","Data":"10dfa9fa10ab17bd5e104fd2dfbeaf1b71842ed08af6a4fadf94866d8f4d03f2"} Nov 01 20:29:12.522542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:12.522482 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" podStartSLOduration=2.522468158 podStartE2EDuration="2.522468158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:12.520226206 +0000 UTC m=+1560.194485461" watchObservedRunningTime="2026-11-01 20:29:12.522468158 +0000 UTC m=+1560.196727412" Nov 01 20:29:13.260589 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:13.260553 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:14.245714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:14.245673 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:16.104606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:16.104574 5633 scope.go:122] "RemoveContainer" containerID="108106c0a4cffd1a45e91728d205646e9f24904ec2cae786ecbbac2f3ebdb963" Nov 01 20:29:16.115746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:16.115720 5633 scope.go:122] "RemoveContainer" containerID="c8582737cede57be225c1567b883c5bd566b739aadeccb62c24c0e5a00a26bf2" Nov 01 20:29:16.126131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:16.126109 5633 scope.go:122] "RemoveContainer" containerID="ab2bf75ae22a399343345ba6625b8267ef2a185ba941b9261cb4fc1b22bc1006" Nov 01 20:29:16.162767 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:29:16.162722 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:29:17.047160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:17.047127 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg"] Nov 01 20:29:17.096842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:17.096812 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ccd6b38-6j6xg"] Nov 01 20:29:17.657358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:17.646375 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:29:18.471758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:18.471719 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3633cce-238d-4572-a3fe-5ce6ee71db3f" path="/var/lib/kubelet/pods/f3633cce-238d-4572-a3fe-5ce6ee71db3f/volumes" Nov 01 20:29:19.121358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:19.118438 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:21.261232 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:21.261194 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:21.274501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:21.274472 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:21.538870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:21.538774 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:29:23.841231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:23.841140 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:24.773558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:24.773508 5633 patch_prober.go:29] interesting pod/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:29:24.773744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:24.773597 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:29:25.588204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:25.588172 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:25.766146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:25.766106 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:26.272869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:26.272838 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:27.578838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:27.578803 5633 generic.go:356] "Generic (PLEG): container finished" podID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerID="39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0" exitCode=0 Nov 01 20:29:27.579516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:27.578879 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" event={"ID":"60e9fdd8-9d3f-44be-8d60-bd816153cf6d","Type":"ContainerDied","Data":"39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0"} Nov 01 20:29:27.579516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:27.579427 5633 scope.go:122] "RemoveContainer" containerID="39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0" Nov 01 20:29:27.581248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:27.581218 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:29:27.581379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:27.581289 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:29:27.581379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:27.581361 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:29:27.947617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:27.947503 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:28.612975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:28.612932 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" event={"ID":"60e9fdd8-9d3f-44be-8d60-bd816153cf6d","Type":"ContainerStarted","Data":"8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20"} Nov 01 20:29:29.448267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:29.448231 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:29.605284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:29.605252 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:29:29.605695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:29.605600 5633 patch_prober.go:29] interesting pod/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 20:29:29.605875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:29.605852 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 20:29:29.707446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:29.707311 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:29.931053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:29.931014 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:30.018490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:30.018446 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc"] Nov 01 20:29:30.058399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:30.058364 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-sdkcc"] Nov 01 20:29:30.467749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:30.467708 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8" path="/var/lib/kubelet/pods/b92cc3ce-a112-4347-b9d3-80e6b3c1ccc8/volumes" Nov 01 20:29:30.589178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:30.588706 5633 patch_prober.go:29] interesting pod/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 20:29:30.589178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:30.588783 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 20:29:31.600357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:31.594735 5633 patch_prober.go:29] interesting pod/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 20:29:31.600357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:31.594814 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 20:29:32.607800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:32.607761 5633 patch_prober.go:29] interesting pod/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 20:29:32.609567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:32.609533 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 20:29:33.051014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:33.050974 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c"] Nov 01 20:29:33.116440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:33.116395 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ebe6afc-lvx5c"] Nov 01 20:29:33.601114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:33.598812 5633 patch_prober.go:29] interesting pod/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 20:29:33.601114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:33.598893 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 20:29:33.871851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:33.871756 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:34.467772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:34.467734 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea602a2d-1c12-4f45-b0e9-ae7809a5820b" path="/var/lib/kubelet/pods/ea602a2d-1c12-4f45-b0e9-ae7809a5820b/volumes" Nov 01 20:29:34.592037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:34.592006 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:29:34.592539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:34.592526 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:29:34.593358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:34.593324 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:29:34.812879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:34.812784 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:36.051527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:36.051492 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq"] Nov 01 20:29:36.098791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:36.098759 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb76be5-fbrsq"] Nov 01 20:29:36.479449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:36.479411 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25fb041-260d-4ec0-86fe-79cef548655f" path="/var/lib/kubelet/pods/f25fb041-260d-4ec0-86fe-79cef548655f/volumes" Nov 01 20:29:37.427489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:37.427457 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:37.601769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:37.601725 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:39.450689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:39.450654 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:29:39.720227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:39.720193 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:39.977372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:39.977018 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:40.097734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.097699 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:40.147444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.147408 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5"] Nov 01 20:29:40.153223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.153198 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.170531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.170503 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.170678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.170654 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-kube-api-access-stl5x\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.213371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.213301 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-71335882-dockercfg-p9gc4\"" Nov 01 20:29:40.297423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.287937 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-kube-api-access-stl5x\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.297423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.288139 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.321598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.321564 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-71335882\"" Nov 01 20:29:40.384485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.349129 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.384686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.370501 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5"] Nov 01 20:29:40.384686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.384571 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq"] Nov 01 20:29:40.392359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.392068 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq"] Nov 01 20:29:40.392359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.392190 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.408711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.408684 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-kube-api-access-stl5x\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.468234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.468200 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:40.491270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.491228 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqql5\" (UniqueName: \"kubernetes.io/projected/34d9d461-cfeb-40fa-8459-e14a64b5e49d-kube-api-access-hqql5\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.491641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.491618 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34d9d461-cfeb-40fa-8459-e14a64b5e49d-plugins-config-volume\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.592986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.592890 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34d9d461-cfeb-40fa-8459-e14a64b5e49d-plugins-config-volume\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.592986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.592979 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqql5\" (UniqueName: \"kubernetes.io/projected/34d9d461-cfeb-40fa-8459-e14a64b5e49d-kube-api-access-hqql5\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.593758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.593735 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34d9d461-cfeb-40fa-8459-e14a64b5e49d-plugins-config-volume\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.650895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.650854 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqql5\" (UniqueName: \"kubernetes.io/projected/34d9d461-cfeb-40fa-8459-e14a64b5e49d-kube-api-access-hqql5\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.711297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.711262 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:40.852935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:40.852908 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5"] Nov 01 20:29:41.047678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.047640 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb"] Nov 01 20:29:41.051437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.051410 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481362b9-jsxrb"] Nov 01 20:29:41.098398 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:29:41.098296 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d9d461_cfeb_40fa_8459_e14a64b5e49d.slice/crio-b42029fe51ede968e9f95718e23d84f4ccc1f75f725589149aadefddcf1fe9ca WatchSource:0}: Error finding container b42029fe51ede968e9f95718e23d84f4ccc1f75f725589149aadefddcf1fe9ca: Status 404 returned error can't find the container with id b42029fe51ede968e9f95718e23d84f4ccc1f75f725589149aadefddcf1fe9ca Nov 01 20:29:41.098687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.098667 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq"] Nov 01 20:29:41.721130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.721091 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" event={"ID":"34d9d461-cfeb-40fa-8459-e14a64b5e49d","Type":"ContainerStarted","Data":"5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1"} Nov 01 20:29:41.721667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.721156 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" event={"ID":"34d9d461-cfeb-40fa-8459-e14a64b5e49d","Type":"ContainerStarted","Data":"b42029fe51ede968e9f95718e23d84f4ccc1f75f725589149aadefddcf1fe9ca"} Nov 01 20:29:41.721667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.721191 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:41.723006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.722981 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" event={"ID":"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0","Type":"ContainerStarted","Data":"9998dffd00fd6961364252b9ccafc24439560f5c343b88ae03a9a5d01be111bc"} Nov 01 20:29:41.723094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.723024 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" event={"ID":"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0","Type":"ContainerStarted","Data":"75ebe3fc9efc201c55c8e26d929417be0418711ed9a584514fe3d645b8f42810"} Nov 01 20:29:41.723094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.723061 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:41.837488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.837433 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" podStartSLOduration=1.837412734 podStartE2EDuration="1.837412734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:41.83363654 +0000 UTC m=+1589.507895826" watchObservedRunningTime="2026-11-01 20:29:41.837412734 +0000 UTC m=+1589.511671988" Nov 01 20:29:41.958933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:41.958880 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" podStartSLOduration=1.958863864 podStartE2EDuration="1.958863864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:41.946725973 +0000 UTC m=+1589.620985248" watchObservedRunningTime="2026-11-01 20:29:41.958863864 +0000 UTC m=+1589.633123118" Nov 01 20:29:42.470425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:42.470390 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac7d07b-208d-4f79-a1bb-2ca39326acd7" path="/var/lib/kubelet/pods/4ac7d07b-208d-4f79-a1bb-2ca39326acd7/volumes" Nov 01 20:29:45.014856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:45.014815 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:46.157158 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:29:46.157117 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:29:47.048382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:47.048348 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m"] Nov 01 20:29:47.051931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:47.051910 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e633ffcf-j5r4m"] Nov 01 20:29:48.467655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:48.467625 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446c9dd3-226b-46b1-a7e8-3505a15d6dac" path="/var/lib/kubelet/pods/446c9dd3-226b-46b1-a7e8-3505a15d6dac/volumes" Nov 01 20:29:49.729748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:49.729695 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:50.046927 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:50.046835 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c"] Nov 01 20:29:50.050370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:50.050343 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5635f84-rcg8c"] Nov 01 20:29:50.469202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:50.469166 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67ca763-e03f-4def-a87c-800b5f434194" path="/var/lib/kubelet/pods/a67ca763-e03f-4def-a87c-800b5f434194/volumes" Nov 01 20:29:52.734764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:52.734729 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:29:52.735238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:52.734802 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:29:52.947602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:52.947563 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:53.076617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:53.076535 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:53.202295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:53.202258 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:54.482136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:54.482092 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:57.757797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.757749 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t"] Nov 01 20:29:57.763754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.763731 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.818987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.818955 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t"] Nov 01 20:29:57.832532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.832493 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.832737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.832707 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.832848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.832823 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphlr\" (UniqueName: \"kubernetes.io/projected/501a9878-6fe4-4e55-a5d9-e9e825520f85-kube-api-access-xphlr\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.862598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.862565 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:29:57.862598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.862580 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:29:57.864732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.864714 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:29:57.864794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.864729 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:29:57.934721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.934683 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.934897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.934764 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphlr\" (UniqueName: \"kubernetes.io/projected/501a9878-6fe4-4e55-a5d9-e9e825520f85-kube-api-access-xphlr\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.934942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.934898 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.935416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.935397 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.937448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.937431 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:57.991626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:57.991595 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphlr\" (UniqueName: \"kubernetes.io/projected/501a9878-6fe4-4e55-a5d9-e9e825520f85-kube-api-access-xphlr\") pod \"maas-api-revoke-keys-e2e-ait-7387069c-2k44t\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:58.079627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:58.079554 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:29:58.462110 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:29:58.461638 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501a9878_6fe4_4e55_a5d9_e9e825520f85.slice/crio-2d94d0fde6f4fc4a035560d03e0c7a4bea13bcfbfb1818b0d3e2ce6dc437c0e8 WatchSource:0}: Error finding container 2d94d0fde6f4fc4a035560d03e0c7a4bea13bcfbfb1818b0d3e2ce6dc437c0e8: Status 404 returned error can't find the container with id 2d94d0fde6f4fc4a035560d03e0c7a4bea13bcfbfb1818b0d3e2ce6dc437c0e8 Nov 01 20:29:58.480227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:58.480203 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t"] Nov 01 20:29:58.888799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:58.888739 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" event={"ID":"501a9878-6fe4-4e55-a5d9-e9e825520f85","Type":"ContainerStarted","Data":"9c277a755571bac546b951e4197838a0d9e7630f70df912d215105809715ac56"} Nov 01 20:29:58.889252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:58.888808 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" event={"ID":"501a9878-6fe4-4e55-a5d9-e9e825520f85","Type":"ContainerStarted","Data":"2d94d0fde6f4fc4a035560d03e0c7a4bea13bcfbfb1818b0d3e2ce6dc437c0e8"} Nov 01 20:29:59.004217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:59.004146 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" podStartSLOduration=2.004129175 podStartE2EDuration="2.004129175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:58.999910669 +0000 UTC m=+1606.674169927" watchObservedRunningTime="2026-11-01 20:29:59.004129175 +0000 UTC m=+1606.678388430" Nov 01 20:29:59.478298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:59.478266 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:29:59.902882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:59.902846 5633 generic.go:356] "Generic (PLEG): container finished" podID="501a9878-6fe4-4e55-a5d9-e9e825520f85" containerID="9c277a755571bac546b951e4197838a0d9e7630f70df912d215105809715ac56" exitCode=0 Nov 01 20:29:59.903278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:29:59.902931 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" event={"ID":"501a9878-6fe4-4e55-a5d9-e9e825520f85","Type":"ContainerDied","Data":"9c277a755571bac546b951e4197838a0d9e7630f70df912d215105809715ac56"} Nov 01 20:30:00.169412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.169291 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw"] Nov 01 20:30:00.175181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.175151 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw"] Nov 01 20:30:00.175358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.175285 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.180290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.180263 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr"] Nov 01 20:30:00.180487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.180380 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.185610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.185570 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6"] Nov 01 20:30:00.185748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.185733 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.190740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.190706 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.191027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.191007 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4"] Nov 01 20:30:00.195520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.195502 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.268144 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268112 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854f3e99-2eeb-4fcd-945d-5660136f7c50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.268355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268181 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2p4m\" (UniqueName: \"kubernetes.io/projected/854f3e99-2eeb-4fcd-945d-5660136f7c50-kube-api-access-r2p4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.268446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268385 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sjmp\" (UniqueName: \"kubernetes.io/projected/ba5c2655-9269-4f2d-aa39-c6d07ac38265-kube-api-access-4sjmp\") pod \"maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.268499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268450 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e69cc-9577-4bf4-8387-923592ab6cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.268559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268533 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-kube-api-access-b77gw\") pod \"maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.268603 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268580 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.268656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268642 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sms8k\" (UniqueName: \"kubernetes.io/projected/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-kube-api-access-sms8k\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.268705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268684 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.268809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268793 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rpb\" (UniqueName: \"kubernetes.io/projected/fa1e69cc-9577-4bf4-8387-923592ab6cac-kube-api-access-g4rpb\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.268853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.268833 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c2655-9269-4f2d-aa39-c6d07ac38265-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.324862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.324819 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw"] Nov 01 20:30:00.325058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.324909 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr"] Nov 01 20:30:00.332448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.332411 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6"] Nov 01 20:30:00.337726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.337696 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4"] Nov 01 20:30:00.369884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.369848 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rpb\" (UniqueName: \"kubernetes.io/projected/fa1e69cc-9577-4bf4-8387-923592ab6cac-kube-api-access-g4rpb\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.370051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.369894 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c2655-9269-4f2d-aa39-c6d07ac38265-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.370051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.369974 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854f3e99-2eeb-4fcd-945d-5660136f7c50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.370051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370034 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2p4m\" (UniqueName: \"kubernetes.io/projected/854f3e99-2eeb-4fcd-945d-5660136f7c50-kube-api-access-r2p4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.370160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370096 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sjmp\" (UniqueName: \"kubernetes.io/projected/ba5c2655-9269-4f2d-aa39-c6d07ac38265-kube-api-access-4sjmp\") pod \"maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.370160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370135 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e69cc-9577-4bf4-8387-923592ab6cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.370241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370166 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-kube-api-access-b77gw\") pod \"maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.370241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370192 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.370241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370219 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sms8k\" (UniqueName: \"kubernetes.io/projected/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-kube-api-access-sms8k\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.370449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370247 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.370795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370765 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c2655-9269-4f2d-aa39-c6d07ac38265-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.370916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370826 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854f3e99-2eeb-4fcd-945d-5660136f7c50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.370916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370884 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e69cc-9577-4bf4-8387-923592ab6cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.371054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.370999 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.371139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.371117 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.380898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.380865 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw"] Nov 01 20:30:00.426467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.426377 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rpb\" (UniqueName: \"kubernetes.io/projected/fa1e69cc-9577-4bf4-8387-923592ab6cac-kube-api-access-g4rpb\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.426631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.426482 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2p4m\" (UniqueName: \"kubernetes.io/projected/854f3e99-2eeb-4fcd-945d-5660136f7c50-kube-api-access-r2p4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.426631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.426556 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sjmp\" (UniqueName: \"kubernetes.io/projected/ba5c2655-9269-4f2d-aa39-c6d07ac38265-kube-api-access-4sjmp\") pod \"maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.426631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.426611 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-kube-api-access-b77gw\") pod \"maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.429075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.429052 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sms8k\" (UniqueName: \"kubernetes.io/projected/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-kube-api-access-sms8k\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.493583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.493548 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:00.502195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.502171 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:00.511867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.511841 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:00.522293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.522263 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:00.527112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.527080 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:00.793404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.793366 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv"] Nov 01 20:30:00.852529 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.852488 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:00.892692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.891768 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw"] Nov 01 20:30:00.893565 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:00.893503 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1e69cc_9577_4bf4_8387_923592ab6cac.slice/crio-d3418e46d58e63b506ff8ec6e64b7015d73070b8f30d9ddeaf8f38d00101abb1 WatchSource:0}: Error finding container d3418e46d58e63b506ff8ec6e64b7015d73070b8f30d9ddeaf8f38d00101abb1: Status 404 returned error can't find the container with id d3418e46d58e63b506ff8ec6e64b7015d73070b8f30d9ddeaf8f38d00101abb1 Nov 01 20:30:00.981897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.981507 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw"] Nov 01 20:30:00.989590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.988763 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr"] Nov 01 20:30:00.995042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:00.995002 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6"] Nov 01 20:30:01.014773 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.002196 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerStarted","Data":"d3418e46d58e63b506ff8ec6e64b7015d73070b8f30d9ddeaf8f38d00101abb1"} Nov 01 20:30:01.016318 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:01.016265 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb1f2c0_57a0_40ac_9d98_8835b526cdea.slice/crio-27fa284b61ce72d4307bf660f4e03702445f680d1c485ead4770520c95b81a12 WatchSource:0}: Error finding container 27fa284b61ce72d4307bf660f4e03702445f680d1c485ead4770520c95b81a12: Status 404 returned error can't find the container with id 27fa284b61ce72d4307bf660f4e03702445f680d1c485ead4770520c95b81a12 Nov 01 20:30:01.023562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.021356 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4"] Nov 01 20:30:01.032138 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:01.030431 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e1c98d_a8aa_46ab_99c5_0694b91cf48e.slice/crio-15f33d9a7857c2d50a52cc250aa2378f078e27e6a23688dbbd95870336dbdca0 WatchSource:0}: Error finding container 15f33d9a7857c2d50a52cc250aa2378f078e27e6a23688dbbd95870336dbdca0: Status 404 returned error can't find the container with id 15f33d9a7857c2d50a52cc250aa2378f078e27e6a23688dbbd95870336dbdca0 Nov 01 20:30:01.255398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.255260 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:01.309378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.309353 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:30:01.388190 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.388140 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-xks-ca\") pod \"501a9878-6fe4-4e55-a5d9-e9e825520f85\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " Nov 01 20:30:01.388278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.388260 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-openshift-ca\") pod \"501a9878-6fe4-4e55-a5d9-e9e825520f85\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " Nov 01 20:30:01.388386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.388372 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501a9878-6fe4-4e55-a5d9-e9e825520f85-kube-api-access-xphlr\" (UniqueName: \"kubernetes.io/projected/501a9878-6fe4-4e55-a5d9-e9e825520f85-kube-api-access-xphlr\") pod \"501a9878-6fe4-4e55-a5d9-e9e825520f85\" (UID: \"501a9878-6fe4-4e55-a5d9-e9e825520f85\") " Nov 01 20:30:01.389636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.389602 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-openshift-ca" pod "501a9878-6fe4-4e55-a5d9-e9e825520f85" (UID: "501a9878-6fe4-4e55-a5d9-e9e825520f85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:01.393514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.393491 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-xks-ca" pod "501a9878-6fe4-4e55-a5d9-e9e825520f85" (UID: "501a9878-6fe4-4e55-a5d9-e9e825520f85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:01.393617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.393507 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501a9878-6fe4-4e55-a5d9-e9e825520f85-kube-api-access-xphlr" pod "501a9878-6fe4-4e55-a5d9-e9e825520f85" (UID: "501a9878-6fe4-4e55-a5d9-e9e825520f85"). InnerVolumeSpecName "kube-api-access-xphlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:01.491078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.491048 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:01.491078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.491077 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501a9878-6fe4-4e55-a5d9-e9e825520f85-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:01.491270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:01.491119 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xphlr\" (UniqueName: \"kubernetes.io/projected/501a9878-6fe4-4e55-a5d9-e9e825520f85-kube-api-access-xphlr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:02.015372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.015176 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerStarted","Data":"0ef53be0b4688982fdef4a1c799938f7ed5a43179fdc49e0250b3f57094016b1"} Nov 01 20:30:02.018995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.018957 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerStarted","Data":"db576b6d6957899e7c9502686ae69969a1f1ec63d6ba81d92ee09dcb0fcda668"} Nov 01 20:30:02.019164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.019021 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerStarted","Data":"7503d6bcf711d807502757fb31c751c76169d66b3fc35087fb4d104f665622d5"} Nov 01 20:30:02.020898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.020868 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerStarted","Data":"f1afd5ff49c6fcfd92ed24a93e90f77dcec2863445425b31f4783d820d61e68e"} Nov 01 20:30:02.021053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.020916 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerStarted","Data":"15f33d9a7857c2d50a52cc250aa2378f078e27e6a23688dbbd95870336dbdca0"} Nov 01 20:30:02.022602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.022574 5633 generic.go:356] "Generic (PLEG): container finished" podID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerID="27a849c0951e43fcbaae06d87c256f8c378c2e4f0a33743e8b2996cc7dcdfac3" exitCode=60 Nov 01 20:30:02.022745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.022660 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" event={"ID":"ba5c2655-9269-4f2d-aa39-c6d07ac38265","Type":"ContainerDied","Data":"27a849c0951e43fcbaae06d87c256f8c378c2e4f0a33743e8b2996cc7dcdfac3"} Nov 01 20:30:02.022745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.022708 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" event={"ID":"ba5c2655-9269-4f2d-aa39-c6d07ac38265","Type":"ContainerStarted","Data":"48955520e6d90c00a72fa2de88b5af31e22f3d30884ed7f5ff0e35625781339e"} Nov 01 20:30:02.022962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.022945 5633 scope.go:122] "RemoveContainer" containerID="27a849c0951e43fcbaae06d87c256f8c378c2e4f0a33743e8b2996cc7dcdfac3" Nov 01 20:30:02.028170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.028141 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerStarted","Data":"fb3fe78e43d1400bbbbb59ea363c15ac4a3172575300002c18a29f8f966aac1b"} Nov 01 20:30:02.028266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.028194 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerStarted","Data":"27fa284b61ce72d4307bf660f4e03702445f680d1c485ead4770520c95b81a12"} Nov 01 20:30:02.030241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.030165 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" event={"ID":"501a9878-6fe4-4e55-a5d9-e9e825520f85","Type":"ContainerDied","Data":"2d94d0fde6f4fc4a035560d03e0c7a4bea13bcfbfb1818b0d3e2ce6dc437c0e8"} Nov 01 20:30:02.030241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.030219 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d94d0fde6f4fc4a035560d03e0c7a4bea13bcfbfb1818b0d3e2ce6dc437c0e8" Nov 01 20:30:02.030241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.030226 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t" Nov 01 20:30:02.132061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.132005 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" podStartSLOduration=2.131988631 podStartE2EDuration="2.131988631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:02.127466684 +0000 UTC m=+1609.801725941" watchObservedRunningTime="2026-11-01 20:30:02.131988631 +0000 UTC m=+1609.806247884" Nov 01 20:30:02.234455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.234419 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:30:02.410372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.405243 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" podStartSLOduration=2.405224649 podStartE2EDuration="2.405224649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:02.40381377 +0000 UTC m=+1610.078073027" watchObservedRunningTime="2026-11-01 20:30:02.405224649 +0000 UTC m=+1610.079483905" Nov 01 20:30:02.520370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.520297 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" podStartSLOduration=2.520282244 podStartE2EDuration="2.520282244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:02.515908985 +0000 UTC m=+1610.190168242" watchObservedRunningTime="2026-11-01 20:30:02.520282244 +0000 UTC m=+1610.194541498" Nov 01 20:30:02.633243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:02.633173 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" podStartSLOduration=2.633152958 podStartE2EDuration="2.633152958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:02.626481389 +0000 UTC m=+1610.300740644" watchObservedRunningTime="2026-11-01 20:30:02.633152958 +0000 UTC m=+1610.307412214" Nov 01 20:30:03.062903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.062865 5633 generic.go:356] "Generic (PLEG): container finished" podID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerID="0ef53be0b4688982fdef4a1c799938f7ed5a43179fdc49e0250b3f57094016b1" exitCode=60 Nov 01 20:30:03.063400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.062986 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerDied","Data":"0ef53be0b4688982fdef4a1c799938f7ed5a43179fdc49e0250b3f57094016b1"} Nov 01 20:30:03.063400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.063317 5633 scope.go:122] "RemoveContainer" containerID="0ef53be0b4688982fdef4a1c799938f7ed5a43179fdc49e0250b3f57094016b1" Nov 01 20:30:03.067579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.067112 5633 generic.go:356] "Generic (PLEG): container finished" podID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerID="db576b6d6957899e7c9502686ae69969a1f1ec63d6ba81d92ee09dcb0fcda668" exitCode=60 Nov 01 20:30:03.067579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.067238 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerDied","Data":"db576b6d6957899e7c9502686ae69969a1f1ec63d6ba81d92ee09dcb0fcda668"} Nov 01 20:30:03.067696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.067640 5633 scope.go:122] "RemoveContainer" containerID="db576b6d6957899e7c9502686ae69969a1f1ec63d6ba81d92ee09dcb0fcda668" Nov 01 20:30:03.069579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.069553 5633 generic.go:356] "Generic (PLEG): container finished" podID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerID="f1afd5ff49c6fcfd92ed24a93e90f77dcec2863445425b31f4783d820d61e68e" exitCode=60 Nov 01 20:30:03.069694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.069677 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerDied","Data":"f1afd5ff49c6fcfd92ed24a93e90f77dcec2863445425b31f4783d820d61e68e"} Nov 01 20:30:03.069908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.069889 5633 scope.go:122] "RemoveContainer" containerID="f1afd5ff49c6fcfd92ed24a93e90f77dcec2863445425b31f4783d820d61e68e" Nov 01 20:30:03.071967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.071942 5633 generic.go:356] "Generic (PLEG): container finished" podID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerID="a870804b9ea45f2dae5f013655b35bb5042bbdf2f76b97980d6ac73f005105f3" exitCode=60 Nov 01 20:30:03.072254 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.072216 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" event={"ID":"ba5c2655-9269-4f2d-aa39-c6d07ac38265","Type":"ContainerDied","Data":"a870804b9ea45f2dae5f013655b35bb5042bbdf2f76b97980d6ac73f005105f3"} Nov 01 20:30:03.072430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.072271 5633 scope.go:122] "RemoveContainer" containerID="27a849c0951e43fcbaae06d87c256f8c378c2e4f0a33743e8b2996cc7dcdfac3" Nov 01 20:30:03.072430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.072370 5633 scope.go:122] "RemoveContainer" containerID="a870804b9ea45f2dae5f013655b35bb5042bbdf2f76b97980d6ac73f005105f3" Nov 01 20:30:03.072721 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:03.072698 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr_odh-ai-gateway-infra(ba5c2655-9269-4f2d-aa39-c6d07ac38265)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" Nov 01 20:30:03.074299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.074274 5633 generic.go:356] "Generic (PLEG): container finished" podID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerID="fb3fe78e43d1400bbbbb59ea363c15ac4a3172575300002c18a29f8f966aac1b" exitCode=60 Nov 01 20:30:03.074421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.074357 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerDied","Data":"fb3fe78e43d1400bbbbb59ea363c15ac4a3172575300002c18a29f8f966aac1b"} Nov 01 20:30:03.074545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:03.074531 5633 scope.go:122] "RemoveContainer" containerID="fb3fe78e43d1400bbbbb59ea363c15ac4a3172575300002c18a29f8f966aac1b" Nov 01 20:30:04.057811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.057778 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84"] Nov 01 20:30:04.061749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.061717 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da88ffc-tgb84"] Nov 01 20:30:04.115499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.115462 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerStarted","Data":"498071d51b79f60bdefc6fa6d7de9cf847c76ef355c26f8b98234eda15521cfb"} Nov 01 20:30:04.120083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.120048 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerStarted","Data":"87f80cdebb6c56ea123ca76073397d262374d042171fba9094439eb2cac50400"} Nov 01 20:30:04.126211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.126174 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerStarted","Data":"20ef09025db17bd5eb7f0005f25b5fde8cda3c65b101d6eb95e42ce09640e15d"} Nov 01 20:30:04.129949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.129921 5633 scope.go:122] "RemoveContainer" containerID="a870804b9ea45f2dae5f013655b35bb5042bbdf2f76b97980d6ac73f005105f3" Nov 01 20:30:04.130564 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:04.130534 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr_odh-ai-gateway-infra(ba5c2655-9269-4f2d-aa39-c6d07ac38265)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" Nov 01 20:30:04.134871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.134842 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerStarted","Data":"4151486c7f345a8a0f3c88136e8bccf1a5039c5e86352f502a9bb9569e109aa8"} Nov 01 20:30:04.470704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.470674 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9767fef0-2dde-4b52-9cf4-4239566d750e" path="/var/lib/kubelet/pods/9767fef0-2dde-4b52-9cf4-4239566d750e/volumes" Nov 01 20:30:04.871568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:04.871461 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:05.145412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.145292 5633 generic.go:356] "Generic (PLEG): container finished" podID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerID="498071d51b79f60bdefc6fa6d7de9cf847c76ef355c26f8b98234eda15521cfb" exitCode=60 Nov 01 20:30:05.145872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.145411 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerDied","Data":"498071d51b79f60bdefc6fa6d7de9cf847c76ef355c26f8b98234eda15521cfb"} Nov 01 20:30:05.145872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.145481 5633 scope.go:122] "RemoveContainer" containerID="0ef53be0b4688982fdef4a1c799938f7ed5a43179fdc49e0250b3f57094016b1" Nov 01 20:30:05.145872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.145825 5633 scope.go:122] "RemoveContainer" containerID="498071d51b79f60bdefc6fa6d7de9cf847c76ef355c26f8b98234eda15521cfb" Nov 01 20:30:05.146198 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:05.146176 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw_odh-ai-gateway-infra(fa1e69cc-9577-4bf4-8387-923592ab6cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" Nov 01 20:30:05.147702 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.147681 5633 generic.go:356] "Generic (PLEG): container finished" podID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerID="87f80cdebb6c56ea123ca76073397d262374d042171fba9094439eb2cac50400" exitCode=60 Nov 01 20:30:05.147805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.147763 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerDied","Data":"87f80cdebb6c56ea123ca76073397d262374d042171fba9094439eb2cac50400"} Nov 01 20:30:05.148024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.148010 5633 scope.go:122] "RemoveContainer" containerID="87f80cdebb6c56ea123ca76073397d262374d042171fba9094439eb2cac50400" Nov 01 20:30:05.148422 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:05.148385 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw_odh-ai-gateway-infra(854f3e99-2eeb-4fcd-945d-5660136f7c50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" Nov 01 20:30:05.149825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.149805 5633 generic.go:356] "Generic (PLEG): container finished" podID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerID="20ef09025db17bd5eb7f0005f25b5fde8cda3c65b101d6eb95e42ce09640e15d" exitCode=60 Nov 01 20:30:05.149949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.149932 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerDied","Data":"20ef09025db17bd5eb7f0005f25b5fde8cda3c65b101d6eb95e42ce09640e15d"} Nov 01 20:30:05.150238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.150221 5633 scope.go:122] "RemoveContainer" containerID="20ef09025db17bd5eb7f0005f25b5fde8cda3c65b101d6eb95e42ce09640e15d" Nov 01 20:30:05.150583 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:05.150562 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4_odh-ai-gateway-infra(e6e1c98d-a8aa-46ab-99c5-0694b91cf48e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" Nov 01 20:30:05.152021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.151996 5633 generic.go:356] "Generic (PLEG): container finished" podID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerID="4151486c7f345a8a0f3c88136e8bccf1a5039c5e86352f502a9bb9569e109aa8" exitCode=60 Nov 01 20:30:05.152125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.152078 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerDied","Data":"4151486c7f345a8a0f3c88136e8bccf1a5039c5e86352f502a9bb9569e109aa8"} Nov 01 20:30:05.152294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.152280 5633 scope.go:122] "RemoveContainer" containerID="4151486c7f345a8a0f3c88136e8bccf1a5039c5e86352f502a9bb9569e109aa8" Nov 01 20:30:05.152612 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:05.152593 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6_odh-ai-gateway-infra(7cb1f2c0-57a0-40ac-9d98-8835b526cdea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" Nov 01 20:30:05.160123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.160100 5633 scope.go:122] "RemoveContainer" containerID="db576b6d6957899e7c9502686ae69969a1f1ec63d6ba81d92ee09dcb0fcda668" Nov 01 20:30:05.177302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.177274 5633 scope.go:122] "RemoveContainer" containerID="f1afd5ff49c6fcfd92ed24a93e90f77dcec2863445425b31f4783d820d61e68e" Nov 01 20:30:05.194373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:05.194344 5633 scope.go:122] "RemoveContainer" containerID="fb3fe78e43d1400bbbbb59ea363c15ac4a3172575300002c18a29f8f966aac1b" Nov 01 20:30:06.164317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:06.164288 5633 scope.go:122] "RemoveContainer" containerID="20ef09025db17bd5eb7f0005f25b5fde8cda3c65b101d6eb95e42ce09640e15d" Nov 01 20:30:06.164828 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:06.164690 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4_odh-ai-gateway-infra(e6e1c98d-a8aa-46ab-99c5-0694b91cf48e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" Nov 01 20:30:06.165999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:06.165978 5633 scope.go:122] "RemoveContainer" containerID="4151486c7f345a8a0f3c88136e8bccf1a5039c5e86352f502a9bb9569e109aa8" Nov 01 20:30:06.166343 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:06.166300 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6_odh-ai-gateway-infra(7cb1f2c0-57a0-40ac-9d98-8835b526cdea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" Nov 01 20:30:06.167848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:06.167825 5633 scope.go:122] "RemoveContainer" containerID="498071d51b79f60bdefc6fa6d7de9cf847c76ef355c26f8b98234eda15521cfb" Nov 01 20:30:06.168159 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:06.168139 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw_odh-ai-gateway-infra(fa1e69cc-9577-4bf4-8387-923592ab6cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" Nov 01 20:30:06.169470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:06.169451 5633 scope.go:122] "RemoveContainer" containerID="87f80cdebb6c56ea123ca76073397d262374d042171fba9094439eb2cac50400" Nov 01 20:30:06.169717 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:06.169694 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw_odh-ai-gateway-infra(854f3e99-2eeb-4fcd-945d-5660136f7c50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" Nov 01 20:30:06.340859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:06.340820 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:08.130004 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.129963 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k"] Nov 01 20:30:08.130840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.130819 5633 cpu_manager.go:414] "containerMap: removing container" podUID="501a9878-6fe4-4e55-a5d9-e9e825520f85" containerName="revoke-keys" Nov 01 20:30:08.130967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.130842 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501a9878-6fe4-4e55-a5d9-e9e825520f85" containerName="revoke-keys" Nov 01 20:30:08.131059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.131045 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="501a9878-6fe4-4e55-a5d9-e9e825520f85" containerName="revoke-keys" Nov 01 20:30:08.136081 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.136060 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.190925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.190891 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k"] Nov 01 20:30:08.216311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.216277 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:30:08.221000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.220966 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.221167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.221104 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczn6\" (UniqueName: \"kubernetes.io/projected/a60c2919-e2b2-4b14-9af2-16c84bcdf826-kube-api-access-lczn6\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.221265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.221245 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.322647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.322615 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.322853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.322676 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczn6\" (UniqueName: \"kubernetes.io/projected/a60c2919-e2b2-4b14-9af2-16c84bcdf826-kube-api-access-lczn6\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.322853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.322776 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.323399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.323368 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.325536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.325512 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.380086 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.380007 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczn6\" (UniqueName: \"kubernetes.io/projected/a60c2919-e2b2-4b14-9af2-16c84bcdf826-kube-api-access-lczn6\") pod \"maas-api-revoke-sub-716d0712-n2t8k\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.450209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.450164 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:08.828291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:08.828252 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k"] Nov 01 20:30:08.828502 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:08.828475 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60c2919_e2b2_4b14_9af2_16c84bcdf826.slice/crio-4bc8e98582590c517c5b4d3fe78d79b608e7bac60cd987e3ba6808d89d6f0872 WatchSource:0}: Error finding container 4bc8e98582590c517c5b4d3fe78d79b608e7bac60cd987e3ba6808d89d6f0872: Status 404 returned error can't find the container with id 4bc8e98582590c517c5b4d3fe78d79b608e7bac60cd987e3ba6808d89d6f0872 Nov 01 20:30:09.197316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:09.197274 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" event={"ID":"a60c2919-e2b2-4b14-9af2-16c84bcdf826","Type":"ContainerStarted","Data":"666f5ae2e98249e6de1c72343d84632012a884df14e24425dc8f46e2882ba358"} Nov 01 20:30:09.197811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:09.197350 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" event={"ID":"a60c2919-e2b2-4b14-9af2-16c84bcdf826","Type":"ContainerStarted","Data":"4bc8e98582590c517c5b4d3fe78d79b608e7bac60cd987e3ba6808d89d6f0872"} Nov 01 20:30:09.312605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:09.312523 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" podStartSLOduration=1.3125050950000001 podStartE2EDuration="1.312505095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:09.309499876 +0000 UTC m=+1616.983759131" watchObservedRunningTime="2026-11-01 20:30:09.312505095 +0000 UTC m=+1616.986764342" Nov 01 20:30:09.577229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:09.577190 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:09.745646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:09.745603 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:09.864138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:09.864104 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:10.209487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:10.209400 5633 generic.go:356] "Generic (PLEG): container finished" podID="a60c2919-e2b2-4b14-9af2-16c84bcdf826" containerID="666f5ae2e98249e6de1c72343d84632012a884df14e24425dc8f46e2882ba358" exitCode=0 Nov 01 20:30:10.209848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:10.209531 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" event={"ID":"a60c2919-e2b2-4b14-9af2-16c84bcdf826","Type":"ContainerDied","Data":"666f5ae2e98249e6de1c72343d84632012a884df14e24425dc8f46e2882ba358"} Nov 01 20:30:11.386543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.386518 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:11.490187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.490095 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60c2919-e2b2-4b14-9af2-16c84bcdf826-kube-api-access-lczn6\" (UniqueName: \"kubernetes.io/projected/a60c2919-e2b2-4b14-9af2-16c84bcdf826-kube-api-access-lczn6\") pod \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " Nov 01 20:30:11.490387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.490195 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-xks-ca\") pod \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " Nov 01 20:30:11.490387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.490273 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-openshift-ca\") pod \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\" (UID: \"a60c2919-e2b2-4b14-9af2-16c84bcdf826\") " Nov 01 20:30:11.490729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.490695 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-openshift-ca" pod "a60c2919-e2b2-4b14-9af2-16c84bcdf826" (UID: "a60c2919-e2b2-4b14-9af2-16c84bcdf826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:11.491377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.491101 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:11.492723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.492698 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-xks-ca" pod "a60c2919-e2b2-4b14-9af2-16c84bcdf826" (UID: "a60c2919-e2b2-4b14-9af2-16c84bcdf826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:11.492882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.492868 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60c2919-e2b2-4b14-9af2-16c84bcdf826-kube-api-access-lczn6" pod "a60c2919-e2b2-4b14-9af2-16c84bcdf826" (UID: "a60c2919-e2b2-4b14-9af2-16c84bcdf826"). InnerVolumeSpecName "kube-api-access-lczn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:11.592964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.592923 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60c2919-e2b2-4b14-9af2-16c84bcdf826-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:11.592964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:11.592961 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczn6\" (UniqueName: \"kubernetes.io/projected/a60c2919-e2b2-4b14-9af2-16c84bcdf826-kube-api-access-lczn6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:12.227567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:12.227531 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" Nov 01 20:30:12.227567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:12.227561 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k" event={"ID":"a60c2919-e2b2-4b14-9af2-16c84bcdf826","Type":"ContainerDied","Data":"4bc8e98582590c517c5b4d3fe78d79b608e7bac60cd987e3ba6808d89d6f0872"} Nov 01 20:30:12.227809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:12.227613 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bc8e98582590c517c5b4d3fe78d79b608e7bac60cd987e3ba6808d89d6f0872" Nov 01 20:30:14.556851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:14.556807 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:14.685980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:14.685943 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:15.460058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:15.460030 5633 scope.go:122] "RemoveContainer" containerID="a870804b9ea45f2dae5f013655b35bb5042bbdf2f76b97980d6ac73f005105f3" Nov 01 20:30:15.824740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:15.824695 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" podUID="2bd8961e-6222-49dd-adcc-d0925e90732a" containerName="main" containerID="cri-o://01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e" gracePeriod=45 Nov 01 20:30:16.121805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.121778 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:30:16.157271 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:16.157235 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:30:16.182896 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.182857 5633 scope.go:122] "RemoveContainer" containerID="502ed2e6d15f00c23b821cbf553ce818645fdb072f7086ff205d31bd8c299ffc" Nov 01 20:30:16.194746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.194718 5633 scope.go:122] "RemoveContainer" containerID="9812087accf509638554f70b0643903ff055ecd9708b2e1fdc46bac58516f139" Nov 01 20:30:16.205674 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.205653 5633 scope.go:122] "RemoveContainer" containerID="efa2f0a960c8ce0257c11107a101be3fd7f6d4503d4e41cf9edeb1f8974c622d" Nov 01 20:30:16.216078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.216057 5633 scope.go:122] "RemoveContainer" containerID="6a185409f4eb784725d5c9293dd6af3b2b07a69c3aed6de2907d9994e8e48f7e" Nov 01 20:30:16.226749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.226729 5633 scope.go:122] "RemoveContainer" containerID="ecf26408aab4ebad0a3c6057f57cba175731f47ebcb9c43bb81f5522a9ba26d1" Nov 01 20:30:16.257453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.257417 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd8961e-6222-49dd-adcc-d0925e90732a-kube-api-access-m4flj\" (UniqueName: \"kubernetes.io/projected/2bd8961e-6222-49dd-adcc-d0925e90732a-kube-api-access-m4flj\") pod \"2bd8961e-6222-49dd-adcc-d0925e90732a\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " Nov 01 20:30:16.257602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.257502 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-home\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-home\") pod \"2bd8961e-6222-49dd-adcc-d0925e90732a\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " Nov 01 20:30:16.257711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.257693 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-dshm\") pod \"2bd8961e-6222-49dd-adcc-d0925e90732a\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " Nov 01 20:30:16.257806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.257757 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-model-cache\") pod \"2bd8961e-6222-49dd-adcc-d0925e90732a\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " Nov 01 20:30:16.257870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.257855 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-tmp-dir\") pod \"2bd8961e-6222-49dd-adcc-d0925e90732a\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " Nov 01 20:30:16.257924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.257891 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8961e-6222-49dd-adcc-d0925e90732a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd8961e-6222-49dd-adcc-d0925e90732a-tls-certs\") pod \"2bd8961e-6222-49dd-adcc-d0925e90732a\" (UID: \"2bd8961e-6222-49dd-adcc-d0925e90732a\") " Nov 01 20:30:16.258664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.258638 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-model-cache" pod "2bd8961e-6222-49dd-adcc-d0925e90732a" (UID: "2bd8961e-6222-49dd-adcc-d0925e90732a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:16.258775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.258702 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-tmp-dir" pod "2bd8961e-6222-49dd-adcc-d0925e90732a" (UID: "2bd8961e-6222-49dd-adcc-d0925e90732a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:16.258954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.258936 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-home" pod "2bd8961e-6222-49dd-adcc-d0925e90732a" (UID: "2bd8961e-6222-49dd-adcc-d0925e90732a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:16.263072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.263048 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-dshm" pod "2bd8961e-6222-49dd-adcc-d0925e90732a" (UID: "2bd8961e-6222-49dd-adcc-d0925e90732a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:16.263072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.263056 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8961e-6222-49dd-adcc-d0925e90732a-tls-certs" pod "2bd8961e-6222-49dd-adcc-d0925e90732a" (UID: "2bd8961e-6222-49dd-adcc-d0925e90732a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:16.279721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.265918 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd8961e-6222-49dd-adcc-d0925e90732a-kube-api-access-m4flj" pod "2bd8961e-6222-49dd-adcc-d0925e90732a" (UID: "2bd8961e-6222-49dd-adcc-d0925e90732a"). InnerVolumeSpecName "kube-api-access-m4flj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:16.284822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.284782 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" event={"ID":"ba5c2655-9269-4f2d-aa39-c6d07ac38265","Type":"ContainerStarted","Data":"ac6fef8f5b2e700073e98e42fd57129c58cf55d5bc548cf3a372955a72666214"} Nov 01 20:30:16.288522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.288490 5633 generic.go:356] "Generic (PLEG): container finished" podID="2bd8961e-6222-49dd-adcc-d0925e90732a" containerID="01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e" exitCode=0 Nov 01 20:30:16.288667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.288598 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" event={"ID":"2bd8961e-6222-49dd-adcc-d0925e90732a","Type":"ContainerDied","Data":"01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e"} Nov 01 20:30:16.288667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.288629 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" event={"ID":"2bd8961e-6222-49dd-adcc-d0925e90732a","Type":"ContainerDied","Data":"10dfa9fa10ab17bd5e104fd2dfbeaf1b71842ed08af6a4fadf94866d8f4d03f2"} Nov 01 20:30:16.288667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.288649 5633 scope.go:122] "RemoveContainer" containerID="01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e" Nov 01 20:30:16.288842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.288801 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv" Nov 01 20:30:16.317298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.317222 5633 scope.go:122] "RemoveContainer" containerID="01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e" Nov 01 20:30:16.317675 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:16.317550 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e\": container with ID starting with 01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e not found: ID does not exist" containerID="01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e" Nov 01 20:30:16.317675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.317600 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e"} err="failed to get container status \"01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e\": rpc error: code = NotFound desc = could not find container \"01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e\": container with ID starting with 01a859036781a4b2dff43844e50d7be639eece8c7ff1681629025860bd159e9e not found: ID does not exist" Nov 01 20:30:16.360425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.360311 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:16.360425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.360356 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:16.360425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.360365 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:16.360425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.360375 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd8961e-6222-49dd-adcc-d0925e90732a-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:16.360425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.360386 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd8961e-6222-49dd-adcc-d0925e90732a-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:16.360425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.360395 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4flj\" (UniqueName: \"kubernetes.io/projected/2bd8961e-6222-49dd-adcc-d0925e90732a-kube-api-access-m4flj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:16.400176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.400118 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" podStartSLOduration=16.400100608 podStartE2EDuration="16.400100608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:16.398699648 +0000 UTC m=+1624.072958905" watchObservedRunningTime="2026-11-01 20:30:16.400100608 +0000 UTC m=+1624.074359863" Nov 01 20:30:16.459912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.459880 5633 scope.go:122] "RemoveContainer" containerID="20ef09025db17bd5eb7f0005f25b5fde8cda3c65b101d6eb95e42ce09640e15d" Nov 01 20:30:16.460119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.460074 5633 scope.go:122] "RemoveContainer" containerID="4151486c7f345a8a0f3c88136e8bccf1a5039c5e86352f502a9bb9569e109aa8" Nov 01 20:30:16.460286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.460263 5633 scope.go:122] "RemoveContainer" containerID="87f80cdebb6c56ea123ca76073397d262374d042171fba9094439eb2cac50400" Nov 01 20:30:16.460626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.460602 5633 scope.go:122] "RemoveContainer" containerID="498071d51b79f60bdefc6fa6d7de9cf847c76ef355c26f8b98234eda15521cfb" Nov 01 20:30:16.511072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.511038 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv"] Nov 01 20:30:16.515467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:16.515442 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-202d23/autores-model-202d23-kserve-85f8f55d45-twzqv"] Nov 01 20:30:17.325059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.324962 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerStarted","Data":"88eaaea4b8c198bb6f41b4d6d9d8c79364b7049709100fe21ae1e818825d9e11"} Nov 01 20:30:17.327040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.327014 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerStarted","Data":"df15d13fbac5332a0b4e076b8b3a9ceef2a47782b1652f14449d6d96f7158756"} Nov 01 20:30:17.328933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.328912 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerStarted","Data":"3614f00e2355ca2360cd15df012310c64cb722fee6132fe3671f6d410eaa51a6"} Nov 01 20:30:17.330851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.330828 5633 generic.go:356] "Generic (PLEG): container finished" podID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerID="ac6fef8f5b2e700073e98e42fd57129c58cf55d5bc548cf3a372955a72666214" exitCode=60 Nov 01 20:30:17.330951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.330893 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" event={"ID":"ba5c2655-9269-4f2d-aa39-c6d07ac38265","Type":"ContainerDied","Data":"ac6fef8f5b2e700073e98e42fd57129c58cf55d5bc548cf3a372955a72666214"} Nov 01 20:30:17.330951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.330947 5633 scope.go:122] "RemoveContainer" containerID="a870804b9ea45f2dae5f013655b35bb5042bbdf2f76b97980d6ac73f005105f3" Nov 01 20:30:17.331267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.331249 5633 scope.go:122] "RemoveContainer" containerID="ac6fef8f5b2e700073e98e42fd57129c58cf55d5bc548cf3a372955a72666214" Nov 01 20:30:17.332202 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:17.332163 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr_odh-ai-gateway-infra(ba5c2655-9269-4f2d-aa39-c6d07ac38265)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" Nov 01 20:30:17.333761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.333735 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerStarted","Data":"cf5884c8ebe23cd8d267b12d7af9578a819835f07b62ac25ac616978ece29031"} Nov 01 20:30:17.409877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:17.409845 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr"] Nov 01 20:30:18.344362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.344316 5633 generic.go:356] "Generic (PLEG): container finished" podID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerID="88eaaea4b8c198bb6f41b4d6d9d8c79364b7049709100fe21ae1e818825d9e11" exitCode=60 Nov 01 20:30:18.344830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.344392 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerDied","Data":"88eaaea4b8c198bb6f41b4d6d9d8c79364b7049709100fe21ae1e818825d9e11"} Nov 01 20:30:18.344830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.344460 5633 scope.go:122] "RemoveContainer" containerID="498071d51b79f60bdefc6fa6d7de9cf847c76ef355c26f8b98234eda15521cfb" Nov 01 20:30:18.344957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.344852 5633 scope.go:122] "RemoveContainer" containerID="88eaaea4b8c198bb6f41b4d6d9d8c79364b7049709100fe21ae1e818825d9e11" Nov 01 20:30:18.345257 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:18.345236 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw_odh-ai-gateway-infra(fa1e69cc-9577-4bf4-8387-923592ab6cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" Nov 01 20:30:18.347118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.347091 5633 generic.go:356] "Generic (PLEG): container finished" podID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerID="df15d13fbac5332a0b4e076b8b3a9ceef2a47782b1652f14449d6d96f7158756" exitCode=60 Nov 01 20:30:18.347228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.347178 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerDied","Data":"df15d13fbac5332a0b4e076b8b3a9ceef2a47782b1652f14449d6d96f7158756"} Nov 01 20:30:18.347760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.347737 5633 scope.go:122] "RemoveContainer" containerID="df15d13fbac5332a0b4e076b8b3a9ceef2a47782b1652f14449d6d96f7158756" Nov 01 20:30:18.348181 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:18.348124 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw_odh-ai-gateway-infra(854f3e99-2eeb-4fcd-945d-5660136f7c50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" Nov 01 20:30:18.360381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.360300 5633 generic.go:356] "Generic (PLEG): container finished" podID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerID="3614f00e2355ca2360cd15df012310c64cb722fee6132fe3671f6d410eaa51a6" exitCode=60 Nov 01 20:30:18.360641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.360551 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerDied","Data":"3614f00e2355ca2360cd15df012310c64cb722fee6132fe3671f6d410eaa51a6"} Nov 01 20:30:18.361091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.361069 5633 scope.go:122] "RemoveContainer" containerID="3614f00e2355ca2360cd15df012310c64cb722fee6132fe3671f6d410eaa51a6" Nov 01 20:30:18.361483 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:18.361462 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4_odh-ai-gateway-infra(e6e1c98d-a8aa-46ab-99c5-0694b91cf48e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" Nov 01 20:30:18.366325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.366301 5633 generic.go:356] "Generic (PLEG): container finished" podID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerID="cf5884c8ebe23cd8d267b12d7af9578a819835f07b62ac25ac616978ece29031" exitCode=60 Nov 01 20:30:18.366461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.366387 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerDied","Data":"cf5884c8ebe23cd8d267b12d7af9578a819835f07b62ac25ac616978ece29031"} Nov 01 20:30:18.366794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.366777 5633 scope.go:122] "RemoveContainer" containerID="cf5884c8ebe23cd8d267b12d7af9578a819835f07b62ac25ac616978ece29031" Nov 01 20:30:18.367118 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:18.367097 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6_odh-ai-gateway-infra(7cb1f2c0-57a0-40ac-9d98-8835b526cdea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" Nov 01 20:30:18.373362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.373307 5633 scope.go:122] "RemoveContainer" containerID="87f80cdebb6c56ea123ca76073397d262374d042171fba9094439eb2cac50400" Nov 01 20:30:18.391438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.391411 5633 scope.go:122] "RemoveContainer" containerID="20ef09025db17bd5eb7f0005f25b5fde8cda3c65b101d6eb95e42ce09640e15d" Nov 01 20:30:18.412050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.412024 5633 scope.go:122] "RemoveContainer" containerID="4151486c7f345a8a0f3c88136e8bccf1a5039c5e86352f502a9bb9569e109aa8" Nov 01 20:30:18.459725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.459682 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw"] Nov 01 20:30:18.478501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.478459 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd8961e-6222-49dd-adcc-d0925e90732a" path="/var/lib/kubelet/pods/2bd8961e-6222-49dd-adcc-d0925e90732a/volumes" Nov 01 20:30:18.563178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.563152 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:18.674393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.674308 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4"] Nov 01 20:30:18.698018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.697975 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5c2655-9269-4f2d-aa39-c6d07ac38265-kube-api-access-4sjmp\" (UniqueName: \"kubernetes.io/projected/ba5c2655-9269-4f2d-aa39-c6d07ac38265-kube-api-access-4sjmp\") pod \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " Nov 01 20:30:18.698202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.698065 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5c2655-9269-4f2d-aa39-c6d07ac38265-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c2655-9269-4f2d-aa39-c6d07ac38265-maas-api-service-ca\") pod \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\" (UID: \"ba5c2655-9269-4f2d-aa39-c6d07ac38265\") " Nov 01 20:30:18.698510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.698482 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5c2655-9269-4f2d-aa39-c6d07ac38265-maas-api-service-ca" pod "ba5c2655-9269-4f2d-aa39-c6d07ac38265" (UID: "ba5c2655-9269-4f2d-aa39-c6d07ac38265"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:18.698687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.698670 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5c2655-9269-4f2d-aa39-c6d07ac38265-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:18.701002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.700980 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5c2655-9269-4f2d-aa39-c6d07ac38265-kube-api-access-4sjmp" pod "ba5c2655-9269-4f2d-aa39-c6d07ac38265" (UID: "ba5c2655-9269-4f2d-aa39-c6d07ac38265"). InnerVolumeSpecName "kube-api-access-4sjmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:18.788698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.788655 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw"] Nov 01 20:30:18.799914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.799877 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sjmp\" (UniqueName: \"kubernetes.io/projected/ba5c2655-9269-4f2d-aa39-c6d07ac38265-kube-api-access-4sjmp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:18.897846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.897811 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6"] Nov 01 20:30:18.935607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.935522 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq"] Nov 01 20:30:18.936208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936194 5633 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8961e-6222-49dd-adcc-d0925e90732a" containerName="main" Nov 01 20:30:18.936246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936209 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8961e-6222-49dd-adcc-d0925e90732a" containerName="main" Nov 01 20:30:18.936246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936228 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:18.936246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936233 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:18.936246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936240 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:18.936246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936245 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:18.936429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936257 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a60c2919-e2b2-4b14-9af2-16c84bcdf826" containerName="revoke-keys" Nov 01 20:30:18.936429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936262 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60c2919-e2b2-4b14-9af2-16c84bcdf826" containerName="revoke-keys" Nov 01 20:30:18.936429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936385 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8961e-6222-49dd-adcc-d0925e90732a" containerName="main" Nov 01 20:30:18.936429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936395 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:18.936429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936409 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:18.936429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.936418 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60c2919-e2b2-4b14-9af2-16c84bcdf826" containerName="revoke-keys" Nov 01 20:30:18.940096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.940073 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:18.980657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.980616 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq"] Nov 01 20:30:18.992052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:18.992015 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:30:19.104354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.104303 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9b8x\" (UniqueName: \"kubernetes.io/projected/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-kube-api-access-b9b8x\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.104517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.104482 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.104645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.104623 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.206914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.206768 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.207086 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.206926 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9b8x\" (UniqueName: \"kubernetes.io/projected/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-kube-api-access-b9b8x\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.207086 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.207037 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.207862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.207805 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.209751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.209722 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.263490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.263450 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9b8x\" (UniqueName: \"kubernetes.io/projected/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-kube-api-access-b9b8x\") pod \"maas-api-revoke-sub-42f896a4-vbnwq\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.374390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.374306 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:19.383717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.383679 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" event={"ID":"ba5c2655-9269-4f2d-aa39-c6d07ac38265","Type":"ContainerDied","Data":"48955520e6d90c00a72fa2de88b5af31e22f3d30884ed7f5ff0e35625781339e"} Nov 01 20:30:19.383717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.383694 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr" Nov 01 20:30:19.383883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.383744 5633 scope.go:122] "RemoveContainer" containerID="ac6fef8f5b2e700073e98e42fd57129c58cf55d5bc548cf3a372955a72666214" Nov 01 20:30:19.555138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.555108 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:19.632680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.632556 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:19.727608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.727079 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-maas-api-service-ca\") pod \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " Nov 01 20:30:19.727608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.727446 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-kube-api-access-sms8k\" (UniqueName: \"kubernetes.io/projected/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-kube-api-access-sms8k\") pod \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\" (UID: \"7cb1f2c0-57a0-40ac-9d98-8835b526cdea\") " Nov 01 20:30:19.727608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.727499 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-maas-api-service-ca" pod "7cb1f2c0-57a0-40ac-9d98-8835b526cdea" (UID: "7cb1f2c0-57a0-40ac-9d98-8835b526cdea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:19.728076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.728022 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:19.734358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.733933 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-kube-api-access-sms8k" pod "7cb1f2c0-57a0-40ac-9d98-8835b526cdea" (UID: "7cb1f2c0-57a0-40ac-9d98-8835b526cdea"). InnerVolumeSpecName "kube-api-access-sms8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:19.761239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.761169 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:19.766306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.766276 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:19.771027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.771000 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:19.829021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.828979 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sms8k\" (UniqueName: \"kubernetes.io/projected/7cb1f2c0-57a0-40ac-9d98-8835b526cdea-kube-api-access-sms8k\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:19.856477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.856440 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr"] Nov 01 20:30:19.860623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.860594 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7387069c-29892750-t7qhr"] Nov 01 20:30:19.930316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.930271 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-maas-api-service-ca\") pod \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " Nov 01 20:30:19.930316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.930319 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-kube-api-access-b77gw\") pod \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\" (UID: \"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e\") " Nov 01 20:30:19.930609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.930527 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854f3e99-2eeb-4fcd-945d-5660136f7c50-kube-api-access-r2p4m\" (UniqueName: \"kubernetes.io/projected/854f3e99-2eeb-4fcd-945d-5660136f7c50-kube-api-access-r2p4m\") pod \"854f3e99-2eeb-4fcd-945d-5660136f7c50\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " Nov 01 20:30:19.930609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.930575 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1e69cc-9577-4bf4-8387-923592ab6cac-kube-api-access-g4rpb\" (UniqueName: \"kubernetes.io/projected/fa1e69cc-9577-4bf4-8387-923592ab6cac-kube-api-access-g4rpb\") pod \"fa1e69cc-9577-4bf4-8387-923592ab6cac\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " Nov 01 20:30:19.930695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.930676 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854f3e99-2eeb-4fcd-945d-5660136f7c50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854f3e99-2eeb-4fcd-945d-5660136f7c50-maas-api-service-ca\") pod \"854f3e99-2eeb-4fcd-945d-5660136f7c50\" (UID: \"854f3e99-2eeb-4fcd-945d-5660136f7c50\") " Nov 01 20:30:19.930779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.930763 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1e69cc-9577-4bf4-8387-923592ab6cac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e69cc-9577-4bf4-8387-923592ab6cac-maas-api-service-ca\") pod \"fa1e69cc-9577-4bf4-8387-923592ab6cac\" (UID: \"fa1e69cc-9577-4bf4-8387-923592ab6cac\") " Nov 01 20:30:19.930842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.930783 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-maas-api-service-ca" pod "e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" (UID: "e6e1c98d-a8aa-46ab-99c5-0694b91cf48e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:19.931114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.931088 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854f3e99-2eeb-4fcd-945d-5660136f7c50-maas-api-service-ca" pod "854f3e99-2eeb-4fcd-945d-5660136f7c50" (UID: "854f3e99-2eeb-4fcd-945d-5660136f7c50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:19.931321 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.931174 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1e69cc-9577-4bf4-8387-923592ab6cac-maas-api-service-ca" pod "fa1e69cc-9577-4bf4-8387-923592ab6cac" (UID: "fa1e69cc-9577-4bf4-8387-923592ab6cac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:19.931555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.931535 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:19.931607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.931565 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/854f3e99-2eeb-4fcd-945d-5660136f7c50-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:19.931607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.931583 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1e69cc-9577-4bf4-8387-923592ab6cac-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:19.932985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.932963 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854f3e99-2eeb-4fcd-945d-5660136f7c50-kube-api-access-r2p4m" pod "854f3e99-2eeb-4fcd-945d-5660136f7c50" (UID: "854f3e99-2eeb-4fcd-945d-5660136f7c50"). InnerVolumeSpecName "kube-api-access-r2p4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:19.933060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.932980 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-kube-api-access-b77gw" pod "e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" (UID: "e6e1c98d-a8aa-46ab-99c5-0694b91cf48e"). InnerVolumeSpecName "kube-api-access-b77gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:19.933060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.932997 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1e69cc-9577-4bf4-8387-923592ab6cac-kube-api-access-g4rpb" pod "fa1e69cc-9577-4bf4-8387-923592ab6cac" (UID: "fa1e69cc-9577-4bf4-8387-923592ab6cac"). InnerVolumeSpecName "kube-api-access-g4rpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:19.979495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:19.979464 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq"] Nov 01 20:30:19.979631 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:19.979565 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a30213_3dbd_47df_a4ef_0e7836f4c4ce.slice/crio-22497b7667f5b2c94c77b1492f23b357d49a9006b290e44860c80d0d7338cc15 WatchSource:0}: Error finding container 22497b7667f5b2c94c77b1492f23b357d49a9006b290e44860c80d0d7338cc15: Status 404 returned error can't find the container with id 22497b7667f5b2c94c77b1492f23b357d49a9006b290e44860c80d0d7338cc15 Nov 01 20:30:20.032969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.032935 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e-kube-api-access-b77gw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:20.032969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.032967 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2p4m\" (UniqueName: \"kubernetes.io/projected/854f3e99-2eeb-4fcd-945d-5660136f7c50-kube-api-access-r2p4m\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:20.032969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.032979 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4rpb\" (UniqueName: \"kubernetes.io/projected/fa1e69cc-9577-4bf4-8387-923592ab6cac-kube-api-access-g4rpb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:20.048848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.048808 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5"] Nov 01 20:30:20.053259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.053226 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d184280-r2hg5"] Nov 01 20:30:20.396158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.396130 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" Nov 01 20:30:20.396662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.396193 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw" event={"ID":"fa1e69cc-9577-4bf4-8387-923592ab6cac","Type":"ContainerDied","Data":"d3418e46d58e63b506ff8ec6e64b7015d73070b8f30d9ddeaf8f38d00101abb1"} Nov 01 20:30:20.396662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.396297 5633 scope.go:122] "RemoveContainer" containerID="88eaaea4b8c198bb6f41b4d6d9d8c79364b7049709100fe21ae1e818825d9e11" Nov 01 20:30:20.398206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.398178 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" event={"ID":"854f3e99-2eeb-4fcd-945d-5660136f7c50","Type":"ContainerDied","Data":"7503d6bcf711d807502757fb31c751c76169d66b3fc35087fb4d104f665622d5"} Nov 01 20:30:20.398206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.398187 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw" Nov 01 20:30:20.400121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.400099 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" Nov 01 20:30:20.400121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.400107 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4" event={"ID":"e6e1c98d-a8aa-46ab-99c5-0694b91cf48e","Type":"ContainerDied","Data":"15f33d9a7857c2d50a52cc250aa2378f078e27e6a23688dbbd95870336dbdca0"} Nov 01 20:30:20.402919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.402891 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" event={"ID":"01a30213-3dbd-47df-a4ef-0e7836f4c4ce","Type":"ContainerStarted","Data":"22497b7667f5b2c94c77b1492f23b357d49a9006b290e44860c80d0d7338cc15"} Nov 01 20:30:20.410217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.410187 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" event={"ID":"7cb1f2c0-57a0-40ac-9d98-8835b526cdea","Type":"ContainerDied","Data":"27fa284b61ce72d4307bf660f4e03702445f680d1c485ead4770520c95b81a12"} Nov 01 20:30:20.410403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.410201 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6" Nov 01 20:30:20.411923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.411894 5633 scope.go:122] "RemoveContainer" containerID="df15d13fbac5332a0b4e076b8b3a9ceef2a47782b1652f14449d6d96f7158756" Nov 01 20:30:20.431580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.430124 5633 scope.go:122] "RemoveContainer" containerID="3614f00e2355ca2360cd15df012310c64cb722fee6132fe3671f6d410eaa51a6" Nov 01 20:30:20.448163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.445393 5633 scope.go:122] "RemoveContainer" containerID="cf5884c8ebe23cd8d267b12d7af9578a819835f07b62ac25ac616978ece29031" Nov 01 20:30:20.472732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.472699 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e94f44-95fe-4306-bff6-69c7176d9f12" path="/var/lib/kubelet/pods/85e94f44-95fe-4306-bff6-69c7176d9f12/volumes" Nov 01 20:30:20.473188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.473174 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" path="/var/lib/kubelet/pods/ba5c2655-9269-4f2d-aa39-c6d07ac38265/volumes" Nov 01 20:30:20.745241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.745202 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4"] Nov 01 20:30:20.749221 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.749193 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-202d23-29892750-xtwq4"] Nov 01 20:30:20.856669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.856582 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw"] Nov 01 20:30:20.862023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.861991 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892750-b6mrw"] Nov 01 20:30:20.969899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.969854 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw"] Nov 01 20:30:20.973541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:20.973516 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892750-56fpw"] Nov 01 20:30:21.079429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:21.079323 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6"] Nov 01 20:30:21.083288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:21.083255 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892750-ccsv6"] Nov 01 20:30:21.422950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:21.422916 5633 generic.go:356] "Generic (PLEG): container finished" podID="01a30213-3dbd-47df-a4ef-0e7836f4c4ce" containerID="0c89f15c83e4719952dd07d1dba2414f7163ee13ad106e27891cdc3d0da9b3ba" exitCode=0 Nov 01 20:30:21.423408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:21.423013 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" event={"ID":"01a30213-3dbd-47df-a4ef-0e7836f4c4ce","Type":"ContainerDied","Data":"0c89f15c83e4719952dd07d1dba2414f7163ee13ad106e27891cdc3d0da9b3ba"} Nov 01 20:30:22.469427 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.469390 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" path="/var/lib/kubelet/pods/7cb1f2c0-57a0-40ac-9d98-8835b526cdea/volumes" Nov 01 20:30:22.470036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.470014 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" path="/var/lib/kubelet/pods/854f3e99-2eeb-4fcd-945d-5660136f7c50/volumes" Nov 01 20:30:22.470524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.470504 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" path="/var/lib/kubelet/pods/e6e1c98d-a8aa-46ab-99c5-0694b91cf48e/volumes" Nov 01 20:30:22.471298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.471277 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" path="/var/lib/kubelet/pods/fa1e69cc-9577-4bf4-8387-923592ab6cac/volumes" Nov 01 20:30:22.598658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.598633 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:22.771429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.771398 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-xks-ca\") pod \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " Nov 01 20:30:22.771613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.771502 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-kube-api-access-b9b8x\" (UniqueName: \"kubernetes.io/projected/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-kube-api-access-b9b8x\") pod \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " Nov 01 20:30:22.771613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.771555 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-openshift-ca\") pod \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\" (UID: \"01a30213-3dbd-47df-a4ef-0e7836f4c4ce\") " Nov 01 20:30:22.771921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.771896 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-openshift-ca" pod "01a30213-3dbd-47df-a4ef-0e7836f4c4ce" (UID: "01a30213-3dbd-47df-a4ef-0e7836f4c4ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:22.773639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.773619 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-xks-ca" pod "01a30213-3dbd-47df-a4ef-0e7836f4c4ce" (UID: "01a30213-3dbd-47df-a4ef-0e7836f4c4ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:22.773804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.773783 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-kube-api-access-b9b8x" pod "01a30213-3dbd-47df-a4ef-0e7836f4c4ce" (UID: "01a30213-3dbd-47df-a4ef-0e7836f4c4ce"). InnerVolumeSpecName "kube-api-access-b9b8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:22.873485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.873449 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:22.873485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.873482 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:22.873485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:22.873492 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9b8x\" (UniqueName: \"kubernetes.io/projected/01a30213-3dbd-47df-a4ef-0e7836f4c4ce-kube-api-access-b9b8x\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:23.442569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:23.442535 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" Nov 01 20:30:23.442777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:23.442566 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq" event={"ID":"01a30213-3dbd-47df-a4ef-0e7836f4c4ce","Type":"ContainerDied","Data":"22497b7667f5b2c94c77b1492f23b357d49a9006b290e44860c80d0d7338cc15"} Nov 01 20:30:23.442777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:23.442613 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22497b7667f5b2c94c77b1492f23b357d49a9006b290e44860c80d0d7338cc15" Nov 01 20:30:24.054785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:24.054753 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:26.612540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:26.612502 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:28.693248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:28.693211 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:30.522565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.522529 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb"] Nov 01 20:30:30.523151 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523136 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523151 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523162 5633 cpu_manager.go:414] "containerMap: removing container" podUID="01a30213-3dbd-47df-a4ef-0e7836f4c4ce" containerName="revoke-keys" Nov 01 20:30:30.523191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523167 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a30213-3dbd-47df-a4ef-0e7836f4c4ce" containerName="revoke-keys" Nov 01 20:30:30.523191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523176 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:30.523191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523180 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:30.523191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523186 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:30.523191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523191 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523208 5633 cpu_manager.go:414] "containerMap: removing container" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523213 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523222 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523227 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523235 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523240 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523247 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523251 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523259 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523263 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523273 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523278 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523295 5633 cpu_manager.go:414] "containerMap: removing container" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523300 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523399 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523408 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a30213-3dbd-47df-a4ef-0e7836f4c4ce" containerName="revoke-keys" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523414 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523419 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5c2655-9269-4f2d-aa39-c6d07ac38265" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523429 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523434 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523439 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:30.523437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523444 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb1f2c0-57a0-40ac-9d98-8835b526cdea" containerName="cleanup" Nov 01 20:30:30.524050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523451 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:30.524050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523456 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:30.524050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.523464 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:30.526942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.526920 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.576212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.576172 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.576410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.576225 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.576410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.576370 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbx9\" (UniqueName: \"kubernetes.io/projected/b65c914c-b5f9-40ee-af6b-66af24b3fff3-kube-api-access-vfbx9\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.582821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.582787 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb"] Nov 01 20:30:30.626966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.626927 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:30:30.626966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.626946 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:30:30.626966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.626945 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:30:30.628681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.628664 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:30:30.677367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.677299 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.677558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.677431 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.677558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.677487 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbx9\" (UniqueName: \"kubernetes.io/projected/b65c914c-b5f9-40ee-af6b-66af24b3fff3-kube-api-access-vfbx9\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.678096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.678076 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.680183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.680162 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.734002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.733966 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbx9\" (UniqueName: \"kubernetes.io/projected/b65c914c-b5f9-40ee-af6b-66af24b3fff3-kube-api-access-vfbx9\") pod \"maas-api-revoke-sub-f22c3354-t9mkb\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:30.841342 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:30.841245 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:31.223972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:31.223946 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb"] Nov 01 20:30:31.224464 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:31.224434 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65c914c_b5f9_40ee_af6b_66af24b3fff3.slice/crio-77879007fb2965bff1b2a64d27140b1cabd8e0b56e6a1a34c731e1a5652bf721 WatchSource:0}: Error finding container 77879007fb2965bff1b2a64d27140b1cabd8e0b56e6a1a34c731e1a5652bf721: Status 404 returned error can't find the container with id 77879007fb2965bff1b2a64d27140b1cabd8e0b56e6a1a34c731e1a5652bf721 Nov 01 20:30:31.517850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:31.517817 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" event={"ID":"b65c914c-b5f9-40ee-af6b-66af24b3fff3","Type":"ContainerStarted","Data":"69ef635891b0a504ab6fb7a983995b90da7795a34d9382ca33891d946bdcd65c"} Nov 01 20:30:31.518000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:31.517867 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" event={"ID":"b65c914c-b5f9-40ee-af6b-66af24b3fff3","Type":"ContainerStarted","Data":"77879007fb2965bff1b2a64d27140b1cabd8e0b56e6a1a34c731e1a5652bf721"} Nov 01 20:30:31.629325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:31.629269 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" podStartSLOduration=1.629250673 podStartE2EDuration="1.629250673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:31.628779168 +0000 UTC m=+1639.303038423" watchObservedRunningTime="2026-11-01 20:30:31.629250673 +0000 UTC m=+1639.303509929" Nov 01 20:30:32.526738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:32.526702 5633 generic.go:356] "Generic (PLEG): container finished" podID="b65c914c-b5f9-40ee-af6b-66af24b3fff3" containerID="69ef635891b0a504ab6fb7a983995b90da7795a34d9382ca33891d946bdcd65c" exitCode=0 Nov 01 20:30:32.526941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:32.526819 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" event={"ID":"b65c914c-b5f9-40ee-af6b-66af24b3fff3","Type":"ContainerDied","Data":"69ef635891b0a504ab6fb7a983995b90da7795a34d9382ca33891d946bdcd65c"} Nov 01 20:30:33.143762 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:33.143720 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:30:33.395179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:33.395081 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:34.030141 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.030101 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:34.159359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.159121 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-xks-ca\") pod \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " Nov 01 20:30:34.159359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.159358 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-openshift-ca\") pod \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " Nov 01 20:30:34.159908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.159511 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65c914c-b5f9-40ee-af6b-66af24b3fff3-kube-api-access-vfbx9\" (UniqueName: \"kubernetes.io/projected/b65c914c-b5f9-40ee-af6b-66af24b3fff3-kube-api-access-vfbx9\") pod \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\" (UID: \"b65c914c-b5f9-40ee-af6b-66af24b3fff3\") " Nov 01 20:30:34.165521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.161160 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-openshift-ca" pod "b65c914c-b5f9-40ee-af6b-66af24b3fff3" (UID: "b65c914c-b5f9-40ee-af6b-66af24b3fff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:34.165521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.163106 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65c914c-b5f9-40ee-af6b-66af24b3fff3-kube-api-access-vfbx9" pod "b65c914c-b5f9-40ee-af6b-66af24b3fff3" (UID: "b65c914c-b5f9-40ee-af6b-66af24b3fff3"). InnerVolumeSpecName "kube-api-access-vfbx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:34.165521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.163750 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-xks-ca" pod "b65c914c-b5f9-40ee-af6b-66af24b3fff3" (UID: "b65c914c-b5f9-40ee-af6b-66af24b3fff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:34.263686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.263595 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.263686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.263638 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbx9\" (UniqueName: \"kubernetes.io/projected/b65c914c-b5f9-40ee-af6b-66af24b3fff3-kube-api-access-vfbx9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.263686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.263654 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65c914c-b5f9-40ee-af6b-66af24b3fff3-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.580858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.580768 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" event={"ID":"b65c914c-b5f9-40ee-af6b-66af24b3fff3","Type":"ContainerDied","Data":"77879007fb2965bff1b2a64d27140b1cabd8e0b56e6a1a34c731e1a5652bf721"} Nov 01 20:30:34.580858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.580788 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb" Nov 01 20:30:34.580858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:34.580823 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77879007fb2965bff1b2a64d27140b1cabd8e0b56e6a1a34c731e1a5652bf721" Nov 01 20:30:35.049915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:35.049875 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn"] Nov 01 20:30:35.053660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:35.053624 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4751cbb5-c84pn"] Nov 01 20:30:36.468167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:36.468136 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb4c017-b7d9-4f5d-981e-1596ec5718a5" path="/var/lib/kubelet/pods/3fb4c017-b7d9-4f5d-981e-1596ec5718a5/volumes" Nov 01 20:30:37.628165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:37.628112 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:39.219397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:39.219351 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:39.490304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:39.490200 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:39.612554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:39.612404 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:39.730108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:39.730068 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t"] Nov 01 20:30:39.777037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:39.776959 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7387069c-2k44t"] Nov 01 20:30:40.469793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:40.469762 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501a9878-6fe4-4e55-a5d9-e9e825520f85" path="/var/lib/kubelet/pods/501a9878-6fe4-4e55-a5d9-e9e825520f85/volumes" Nov 01 20:30:41.134211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.134174 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:41.152434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.152391 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6"] Nov 01 20:30:41.152633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.152613 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" containerID="cri-o://8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20" gracePeriod=30 Nov 01 20:30:41.598680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.598639 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw"] Nov 01 20:30:41.599286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599273 5633 cpu_manager.go:414] "containerMap: removing container" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:41.599354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599288 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:41.599354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599309 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:41.599354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599314 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:41.599354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599353 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b65c914c-b5f9-40ee-af6b-66af24b3fff3" containerName="revoke-keys" Nov 01 20:30:41.599498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599361 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65c914c-b5f9-40ee-af6b-66af24b3fff3" containerName="revoke-keys" Nov 01 20:30:41.599498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599377 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:41.599498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599382 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:41.599498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599467 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e1c98d-a8aa-46ab-99c5-0694b91cf48e" containerName="cleanup" Nov 01 20:30:41.599498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599479 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65c914c-b5f9-40ee-af6b-66af24b3fff3" containerName="revoke-keys" Nov 01 20:30:41.599498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599487 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1e69cc-9577-4bf4-8387-923592ab6cac" containerName="cleanup" Nov 01 20:30:41.599498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.599495 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="854f3e99-2eeb-4fcd-945d-5660136f7c50" containerName="cleanup" Nov 01 20:30:41.604892 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.604874 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.656740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.656704 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:30:41.656982 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.656914 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:30:41.656982 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.656931 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:30:41.656982 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.656978 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:30:41.658811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.658788 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw"] Nov 01 20:30:41.723020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.722988 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/986f2f35-5197-40d4-943e-ee2bf738ee43-kube-api-access-9q4fb\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.723195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.723104 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.723238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.723191 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.736863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.736811 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:41.824824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.824783 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/986f2f35-5197-40d4-943e-ee2bf738ee43-kube-api-access-9q4fb\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.825028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.824965 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.825094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.825046 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.825829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.825802 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.827744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.827712 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.884754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.884684 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/986f2f35-5197-40d4-943e-ee2bf738ee43-kube-api-access-9q4fb\") pod \"maas-api-revoke-sub-43bc06ec-smlsw\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:41.918726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:41.918695 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:42.236414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.236367 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f"] Nov 01 20:30:42.241388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.241314 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.364378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.363484 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-393dc263-openshift-default-dockercfg-ssfv2\"" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365270 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365314 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365507 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365553 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365629 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365659 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzpt\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-kube-api-access-vbzpt\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365843 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365881 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.365955 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.371929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.366945 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:42.396139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.395403 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f"] Nov 01 20:30:42.403053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.399932 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw"] Nov 01 20:30:42.575464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575423 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.575658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575492 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzpt\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-kube-api-access-vbzpt\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.575735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575709 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.575799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575749 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.575858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575825 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.575916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575878 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.575916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575900 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.576083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.575990 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.576083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.576001 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.576083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.576044 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.576453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.576427 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.576976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.576948 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.577107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.576437 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.577853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.577826 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.579873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.579842 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.590237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.589948 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.633206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.633173 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.633614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.633218 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzpt\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-kube-api-access-vbzpt\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-c259f\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.689051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.689013 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" event={"ID":"986f2f35-5197-40d4-943e-ee2bf738ee43","Type":"ContainerStarted","Data":"65ed96f538a07d9ce04acd27764d55ddc953c7145773618d18ab240d2b5b92b0"} Nov 01 20:30:42.856976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.856944 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:42.931998 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:42.931958 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:43.367418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.367383 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f"] Nov 01 20:30:43.367786 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:43.367760 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8de12b_5edf_4ebe_afbc_980a7d80aeb4.slice/crio-6a2e56e5af05f346d8de6cdd5c988f7f72bff032809bd3c2ef44aa4fa83f107a WatchSource:0}: Error finding container 6a2e56e5af05f346d8de6cdd5c988f7f72bff032809bd3c2ef44aa4fa83f107a: Status 404 returned error can't find the container with id 6a2e56e5af05f346d8de6cdd5c988f7f72bff032809bd3c2ef44aa4fa83f107a Nov 01 20:30:43.370487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.370454 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:30:43.370581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.370520 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:30:43.370581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.370549 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:30:43.701866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.701756 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" event={"ID":"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4","Type":"ContainerStarted","Data":"982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891"} Nov 01 20:30:43.701866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.701831 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" event={"ID":"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4","Type":"ContainerStarted","Data":"6a2e56e5af05f346d8de6cdd5c988f7f72bff032809bd3c2ef44aa4fa83f107a"} Nov 01 20:30:43.704624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.704597 5633 generic.go:356] "Generic (PLEG): container finished" podID="986f2f35-5197-40d4-943e-ee2bf738ee43" containerID="30140ae1d3f72bfca04d0f226b6ca97e0d131e383d4002dc0ece15c881fd55c2" exitCode=0 Nov 01 20:30:43.704787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.704656 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" event={"ID":"986f2f35-5197-40d4-943e-ee2bf738ee43","Type":"ContainerDied","Data":"30140ae1d3f72bfca04d0f226b6ca97e0d131e383d4002dc0ece15c881fd55c2"} Nov 01 20:30:43.752633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.752602 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:43.819885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.819818 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podStartSLOduration=1.81979566 podStartE2EDuration="1.81979566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:43.81645419 +0000 UTC m=+1651.490713446" watchObservedRunningTime="2026-11-01 20:30:43.81979566 +0000 UTC m=+1651.494054916" Nov 01 20:30:43.857619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.857580 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:43.868613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.868579 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:44.000054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:43.999966 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:44.764453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:44.764414 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:44.875311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:44.874954 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:44.959493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:44.959467 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:45.008900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.008820 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-openshift-ca\") pod \"986f2f35-5197-40d4-943e-ee2bf738ee43\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " Nov 01 20:30:45.009063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.008953 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986f2f35-5197-40d4-943e-ee2bf738ee43-kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/986f2f35-5197-40d4-943e-ee2bf738ee43-kube-api-access-9q4fb\") pod \"986f2f35-5197-40d4-943e-ee2bf738ee43\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " Nov 01 20:30:45.009063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.008983 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-xks-ca\") pod \"986f2f35-5197-40d4-943e-ee2bf738ee43\" (UID: \"986f2f35-5197-40d4-943e-ee2bf738ee43\") " Nov 01 20:30:45.009172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.009151 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-openshift-ca" pod "986f2f35-5197-40d4-943e-ee2bf738ee43" (UID: "986f2f35-5197-40d4-943e-ee2bf738ee43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:45.009653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.009623 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:45.011219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.011187 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-xks-ca" pod "986f2f35-5197-40d4-943e-ee2bf738ee43" (UID: "986f2f35-5197-40d4-943e-ee2bf738ee43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:45.011374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.011281 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986f2f35-5197-40d4-943e-ee2bf738ee43-kube-api-access-9q4fb" pod "986f2f35-5197-40d4-943e-ee2bf738ee43" (UID: "986f2f35-5197-40d4-943e-ee2bf738ee43"). InnerVolumeSpecName "kube-api-access-9q4fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:45.111919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.111883 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q4fb\" (UniqueName: \"kubernetes.io/projected/986f2f35-5197-40d4-943e-ee2bf738ee43-kube-api-access-9q4fb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:45.111919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.111918 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/986f2f35-5197-40d4-943e-ee2bf738ee43-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:45.723205 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.723168 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" Nov 01 20:30:45.723449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.723212 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw" event={"ID":"986f2f35-5197-40d4-943e-ee2bf738ee43","Type":"ContainerDied","Data":"65ed96f538a07d9ce04acd27764d55ddc953c7145773618d18ab240d2b5b92b0"} Nov 01 20:30:45.723449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.723257 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ed96f538a07d9ce04acd27764d55ddc953c7145773618d18ab240d2b5b92b0" Nov 01 20:30:45.723582 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.723458 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:45.724459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.724439 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:30:45.829180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:45.829149 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:46.014204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.014118 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:46.147876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.147842 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:46.160040 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:46.160004 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:30:46.459508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.459472 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:30:46.536118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536033 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-token\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-token\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536072 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-envoy\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536201 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-data\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536360 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istiod-ca-cert\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536468 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-data" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:46.536556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536500 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-kube-api-access-2xwhr\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-kube-api-access-2xwhr\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536547 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-podinfo\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536582 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-certs\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536614 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-socket\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.536756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536684 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-credential-socket\") pod \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\" (UID: \"60e9fdd8-9d3f-44be-8d60-bd816153cf6d\") " Nov 01 20:30:46.537041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.536679 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istiod-ca-cert" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:46.537156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.537135 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-certs" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:46.537156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.537006 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-credential-socket" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:46.537919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.537892 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.538054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.537922 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.538054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.537940 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.538054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.537956 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.538054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.537971 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-socket" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:46.540355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.539025 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-envoy" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:46.540355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.539153 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-kube-api-access-2xwhr" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "kube-api-access-2xwhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:46.540355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.539188 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-token" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:46.540355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.539411 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-podinfo" pod "60e9fdd8-9d3f-44be-8d60-bd816153cf6d" (UID: "60e9fdd8-9d3f-44be-8d60-bd816153cf6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:30:46.639391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.639321 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwhr\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-kube-api-access-2xwhr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.639391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.639389 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.639391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.639399 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.639648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.639407 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.639648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.639415 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e9fdd8-9d3f-44be-8d60-bd816153cf6d-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:46.733012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.732965 5633 generic.go:356] "Generic (PLEG): container finished" podID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerID="8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20" exitCode=0 Nov 01 20:30:46.733209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.733034 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" event={"ID":"60e9fdd8-9d3f-44be-8d60-bd816153cf6d","Type":"ContainerDied","Data":"8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20"} Nov 01 20:30:46.733209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.733086 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" Nov 01 20:30:46.733209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.733101 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6" event={"ID":"60e9fdd8-9d3f-44be-8d60-bd816153cf6d","Type":"ContainerDied","Data":"0ae2e85f8ad1de75c12785434a0fcb91d9599109eaf1da99b2d2bc2cd47a7b23"} Nov 01 20:30:46.733209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.733131 5633 scope.go:122] "RemoveContainer" containerID="8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20" Nov 01 20:30:46.746009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.745982 5633 scope.go:122] "RemoveContainer" containerID="39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0" Nov 01 20:30:46.757638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.757617 5633 scope.go:122] "RemoveContainer" containerID="8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20" Nov 01 20:30:46.757968 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:46.757944 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20\": container with ID starting with 8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20 not found: ID does not exist" containerID="8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20" Nov 01 20:30:46.758070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.757978 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20"} err="failed to get container status \"8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20\": rpc error: code = NotFound desc = could not find container \"8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20\": container with ID starting with 8c99e4f6eddf12888202f1344816f905faeef260a08817c8c9d31f23ed995a20 not found: ID does not exist" Nov 01 20:30:46.758070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.757999 5633 scope.go:122] "RemoveContainer" containerID="39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0" Nov 01 20:30:46.758314 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:46.758290 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0\": container with ID starting with 39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0 not found: ID does not exist" containerID="39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0" Nov 01 20:30:46.758399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.758324 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0"} err="failed to get container status \"39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0\": rpc error: code = NotFound desc = could not find container \"39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0\": container with ID starting with 39c407038a79721629f58c5a7e71905f3c381bf2f20df31229cf7520bdfd31b0 not found: ID does not exist" Nov 01 20:30:46.905040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.904958 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6"] Nov 01 20:30:46.908612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:46.908578 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7387069c-openshift-default-5b6698d4dc-qkqr6"] Nov 01 20:30:48.511410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:48.510804 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" path="/var/lib/kubelet/pods/60e9fdd8-9d3f-44be-8d60-bd816153cf6d/volumes" Nov 01 20:30:49.811012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:49.810976 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:50.183873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:50.183768 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:50.351167 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:50.351130 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s"] Nov 01 20:30:50.351506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:50.351480 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" containerID="cri-o://e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178" gracePeriod=30 Nov 01 20:30:50.418044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:50.418007 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2"] Nov 01 20:30:50.419194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:50.419154 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" containerID="cri-o://22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088" gracePeriod=30 Nov 01 20:30:50.424916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:50.424877 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v"] Nov 01 20:30:50.425166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:50.425139 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" containerID="cri-o://89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786" gracePeriod=30 Nov 01 20:30:51.045366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.045313 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb"] Nov 01 20:30:51.049170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.049142 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29856add-djgqb"] Nov 01 20:30:51.299936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.299829 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:51.391144 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.391096 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:30:51.391324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.391164 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:30:51.499255 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.499216 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:51.722846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.722810 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz"] Nov 01 20:30:51.723481 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.723465 5633 cpu_manager.go:414] "containerMap: removing container" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" Nov 01 20:30:51.723539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.723482 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" Nov 01 20:30:51.723539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.723506 5633 cpu_manager.go:414] "containerMap: removing container" podUID="986f2f35-5197-40d4-943e-ee2bf738ee43" containerName="revoke-keys" Nov 01 20:30:51.723539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.723510 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986f2f35-5197-40d4-943e-ee2bf738ee43" containerName="revoke-keys" Nov 01 20:30:51.723636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.723622 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" Nov 01 20:30:51.723670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.723637 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" Nov 01 20:30:51.723670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.723647 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="986f2f35-5197-40d4-943e-ee2bf738ee43" containerName="revoke-keys" Nov 01 20:30:51.728605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.728582 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:51.780156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.780124 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-393dc263\"" Nov 01 20:30:51.780359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.780189 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-393dc263-dockercfg-8xzw9\"" Nov 01 20:30:51.782054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.782026 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz"] Nov 01 20:30:51.902706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.902661 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rcs\" (UniqueName: \"kubernetes.io/projected/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-kube-api-access-k8rcs\") pod \"payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:51.902903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.902780 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:51.917882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.917843 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d"] Nov 01 20:30:51.918689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.918666 5633 cpu_manager.go:414] "containerMap: removing container" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" Nov 01 20:30:51.918689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.918687 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e9fdd8-9d3f-44be-8d60-bd816153cf6d" containerName="istio-proxy" Nov 01 20:30:51.922975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.922950 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:51.939492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:51.939461 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d"] Nov 01 20:30:52.004163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.004069 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rcs\" (UniqueName: \"kubernetes.io/projected/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-kube-api-access-k8rcs\") pod \"payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:52.004377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.004165 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:52.004961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.004937 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:52.060388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.060355 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rcs\" (UniqueName: \"kubernetes.io/projected/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-kube-api-access-k8rcs\") pod \"payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:52.105523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.105483 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xbrz\" (UniqueName: \"kubernetes.io/projected/636e42fc-694a-4c2a-9cfa-b3bff1c05790-kube-api-access-6xbrz\") pod \"payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.105676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.105551 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e42fc-694a-4c2a-9cfa-b3bff1c05790-plugins-config-volume\") pod \"payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.206809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.206773 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbrz\" (UniqueName: \"kubernetes.io/projected/636e42fc-694a-4c2a-9cfa-b3bff1c05790-kube-api-access-6xbrz\") pod \"payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.206992 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.206825 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e42fc-694a-4c2a-9cfa-b3bff1c05790-plugins-config-volume\") pod \"payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.207548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.207527 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e42fc-694a-4c2a-9cfa-b3bff1c05790-plugins-config-volume\") pod \"payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.263237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.263163 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbrz\" (UniqueName: \"kubernetes.io/projected/636e42fc-694a-4c2a-9cfa-b3bff1c05790-kube-api-access-6xbrz\") pod \"payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.277296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.277261 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.343800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.343752 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:52.469062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.469026 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f210ad-a9cf-4d57-98b3-06360393d1a4" path="/var/lib/kubelet/pods/90f210ad-a9cf-4d57-98b3-06360393d1a4/volumes" Nov 01 20:30:52.670584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.670551 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d"] Nov 01 20:30:52.671132 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:52.671081 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636e42fc_694a_4c2a_9cfa_b3bff1c05790.slice/crio-852033edf3258a3fa9abaf88a93783cc911f8123f18be81723c7f8902272a73e WatchSource:0}: Error finding container 852033edf3258a3fa9abaf88a93783cc911f8123f18be81723c7f8902272a73e: Status 404 returned error can't find the container with id 852033edf3258a3fa9abaf88a93783cc911f8123f18be81723c7f8902272a73e Nov 01 20:30:52.780007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.779972 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz"] Nov 01 20:30:52.780348 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:30:52.780301 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52969ce6_85f7_4a62_8204_8c42d0d0c5b6.slice/crio-3c4db9dd73c9e0f2bb9ac114ebbd43d781685771571130cc7f57021cb813a4af WatchSource:0}: Error finding container 3c4db9dd73c9e0f2bb9ac114ebbd43d781685771571130cc7f57021cb813a4af: Status 404 returned error can't find the container with id 3c4db9dd73c9e0f2bb9ac114ebbd43d781685771571130cc7f57021cb813a4af Nov 01 20:30:52.923756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.923663 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" event={"ID":"636e42fc-694a-4c2a-9cfa-b3bff1c05790","Type":"ContainerStarted","Data":"3acc54ab9d9c71c08dd4c0476823c959f6e7d36f36689ffd591d8b964a37b0bf"} Nov 01 20:30:52.923756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.923716 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" event={"ID":"636e42fc-694a-4c2a-9cfa-b3bff1c05790","Type":"ContainerStarted","Data":"852033edf3258a3fa9abaf88a93783cc911f8123f18be81723c7f8902272a73e"} Nov 01 20:30:52.923756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.923747 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:30:52.925591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.925560 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" event={"ID":"52969ce6-85f7-4a62-8204-8c42d0d0c5b6","Type":"ContainerStarted","Data":"b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00"} Nov 01 20:30:52.925691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.925617 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" event={"ID":"52969ce6-85f7-4a62-8204-8c42d0d0c5b6","Type":"ContainerStarted","Data":"3c4db9dd73c9e0f2bb9ac114ebbd43d781685771571130cc7f57021cb813a4af"} Nov 01 20:30:52.925691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:52.925647 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:30:53.040420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:53.040144 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" podStartSLOduration=2.040124604 podStartE2EDuration="2.040124604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:53.036587077 +0000 UTC m=+1660.710846345" watchObservedRunningTime="2026-11-01 20:30:53.040124604 +0000 UTC m=+1660.714383861" Nov 01 20:30:53.159243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:53.159175 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" podStartSLOduration=2.159148216 podStartE2EDuration="2.159148216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:53.152846293 +0000 UTC m=+1660.827105584" watchObservedRunningTime="2026-11-01 20:30:53.159148216 +0000 UTC m=+1660.833407472" Nov 01 20:30:54.135283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:54.135230 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:30:54.135508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:54.135302 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:30:54.511906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:54.511428 5633 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:30:54.511906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:54.511507 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:30:55.069893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.069859 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:55.702353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.702314 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:30:55.740197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.740165 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:30:55.754531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754503 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istiod-ca-cert\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.754754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754546 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-envoy\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.754754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754580 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-certs\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.754754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754625 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-socket\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.754754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754654 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-certs\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.754754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754715 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-podinfo\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.755070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754775 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-kube-api-access-6tpsc\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-kube-api-access-6tpsc\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.755070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754807 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-data\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.755070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754853 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-socket\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.755070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754887 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-envoy\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.755070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.754968 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-data\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.755070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.755004 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-podinfo\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.755070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.755058 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istiod-ca-cert" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:55.755775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.755744 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-socket" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.756402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756266 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-data" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.756402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756382 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-token\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.756658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756474 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-data" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.756658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756481 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-credential-socket\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.756658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756548 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-credential-socket\") pod \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\" (UID: \"437e5a55-57f6-4a8b-a2b0-893259e41ad3\") " Nov 01 20:30:55.756658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756576 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-token\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.756658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756614 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-kube-api-access-fbc82\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-kube-api-access-fbc82\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.756658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.756648 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed39b20-1f38-4a11-a398-ccb791e0a159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed39b20-1f38-4a11-a398-ccb791e0a159-istiod-ca-cert\") pod \"1ed39b20-1f38-4a11-a398-ccb791e0a159\" (UID: \"1ed39b20-1f38-4a11-a398-ccb791e0a159\") " Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757275 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-credential-socket" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757509 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-credential-socket" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757552 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-certs" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757586 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757606 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757622 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757636 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.757697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757650 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.758189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.757988 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-certs" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.758189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.758051 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-socket" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.758309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.758215 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed39b20-1f38-4a11-a398-ccb791e0a159-istiod-ca-cert" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:55.759930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.759902 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-podinfo" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:30:55.760183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.760126 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-podinfo" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:30:55.760183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.760147 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-kube-api-access-6tpsc" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "kube-api-access-6tpsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:55.760347 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.760291 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-token" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:55.761157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.761127 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-kube-api-access-fbc82" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "kube-api-access-fbc82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:55.761284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.761266 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-token" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:55.762393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.762324 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-envoy" pod "1ed39b20-1f38-4a11-a398-ccb791e0a159" (UID: "1ed39b20-1f38-4a11-a398-ccb791e0a159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.763300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.763278 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-envoy" pod "437e5a55-57f6-4a8b-a2b0-893259e41ad3" (UID: "437e5a55-57f6-4a8b-a2b0-893259e41ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.792725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.792696 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:30:55.835587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.835536 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:55.858864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.858835 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istiod-ca-cert\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.858864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.858865 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-token\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-token\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.858925 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-data\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.858979 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-socket\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859016 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-envoy\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859120 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-certs\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859171 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-podinfo\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859233 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-credential-socket\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859260 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istiod-ca-cert" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:55.859299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859268 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-kube-api-access-h28fn\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-kube-api-access-h28fn\") pod \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\" (UID: \"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd\") " Nov 01 20:30:55.859299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859282 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-data" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.859542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859437 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-socket" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.859607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859537 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-credential-socket" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.859669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.859626 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-certs" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.860083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860061 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860090 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860105 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860119 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbc82\" (UniqueName: \"kubernetes.io/projected/1ed39b20-1f38-4a11-a398-ccb791e0a159-kube-api-access-fbc82\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860133 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed39b20-1f38-4a11-a398-ccb791e0a159-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860148 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860161 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860174 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860189 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860203 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860216 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437e5a55-57f6-4a8b-a2b0-893259e41ad3-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860231 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860246 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860259 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tpsc\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-kube-api-access-6tpsc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860272 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed39b20-1f38-4a11-a398-ccb791e0a159-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860286 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860300 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.860446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.860315 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437e5a55-57f6-4a8b-a2b0-893259e41ad3-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.861493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.861472 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-envoy" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:55.861556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.861515 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-token" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:55.862011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.861990 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-podinfo" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:30:55.862112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.862019 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-kube-api-access-h28fn" pod "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" (UID: "08ccf9e0-4a1f-44e9-b032-13d7d763dbdd"). InnerVolumeSpecName "kube-api-access-h28fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:55.952044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.951945 5633 generic.go:356] "Generic (PLEG): container finished" podID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerID="22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088" exitCode=0 Nov 01 20:30:55.952044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.951979 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" event={"ID":"437e5a55-57f6-4a8b-a2b0-893259e41ad3","Type":"ContainerDied","Data":"22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088"} Nov 01 20:30:55.952044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.952024 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" Nov 01 20:30:55.952317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.952046 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2" event={"ID":"437e5a55-57f6-4a8b-a2b0-893259e41ad3","Type":"ContainerDied","Data":"074657a89933be22eeda57c78f5193c665cb345654c3ce51f730b69fb2d424c7"} Nov 01 20:30:55.952317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.952084 5633 scope.go:122] "RemoveContainer" containerID="22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088" Nov 01 20:30:55.953554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.953138 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:30:55.954186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.954044 5633 generic.go:356] "Generic (PLEG): container finished" podID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerID="e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178" exitCode=0 Nov 01 20:30:55.954186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.954069 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" event={"ID":"1ed39b20-1f38-4a11-a398-ccb791e0a159","Type":"ContainerDied","Data":"e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178"} Nov 01 20:30:55.954186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.954097 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" Nov 01 20:30:55.954186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.954115 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s" event={"ID":"1ed39b20-1f38-4a11-a398-ccb791e0a159","Type":"ContainerDied","Data":"5048aa72923deb4e0f797d0ef43aaac558033459785c84935bd114a26fe3aabc"} Nov 01 20:30:55.956325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.956296 5633 generic.go:356] "Generic (PLEG): container finished" podID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerID="89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786" exitCode=0 Nov 01 20:30:55.956537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.956431 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" event={"ID":"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd","Type":"ContainerDied","Data":"89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786"} Nov 01 20:30:55.956537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.956474 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" event={"ID":"08ccf9e0-4a1f-44e9-b032-13d7d763dbdd","Type":"ContainerDied","Data":"7746138cefd08aeb1b7196cd035405f99438ddebc06cafa668789d589c364b49"} Nov 01 20:30:55.956676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.956561 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v" Nov 01 20:30:55.962964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.962934 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.962964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.962962 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28fn\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-kube-api-access-h28fn\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.963180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.962978 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.963180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.962988 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:55.966900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.966882 5633 scope.go:122] "RemoveContainer" containerID="22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088" Nov 01 20:30:55.967212 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:55.967193 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088\": container with ID starting with 22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088 not found: ID does not exist" containerID="22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088" Nov 01 20:30:55.967307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.967221 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088"} err="failed to get container status \"22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088\": rpc error: code = NotFound desc = could not find container \"22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088\": container with ID starting with 22dba90731431eb726746953c62bd59229c73f764e26f138c9afab6fb7d7c088 not found: ID does not exist" Nov 01 20:30:55.967307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:55.967243 5633 scope.go:122] "RemoveContainer" containerID="e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178" Nov 01 20:30:56.006069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.006045 5633 scope.go:122] "RemoveContainer" containerID="e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178" Nov 01 20:30:56.006664 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:56.006499 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178\": container with ID starting with e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178 not found: ID does not exist" containerID="e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178" Nov 01 20:30:56.006664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.006538 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178"} err="failed to get container status \"e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178\": rpc error: code = NotFound desc = could not find container \"e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178\": container with ID starting with e48d155fe3bcf4c5a449eaf722a878204cd05947ec48176bec4d0c73e876e178 not found: ID does not exist" Nov 01 20:30:56.006664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.006564 5633 scope.go:122] "RemoveContainer" containerID="89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786" Nov 01 20:30:56.029782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.029756 5633 scope.go:122] "RemoveContainer" containerID="c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955" Nov 01 20:30:56.050783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.050650 5633 scope.go:122] "RemoveContainer" containerID="89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786" Nov 01 20:30:56.051139 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:56.051104 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786\": container with ID starting with 89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786 not found: ID does not exist" containerID="89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786" Nov 01 20:30:56.051300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.051146 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786"} err="failed to get container status \"89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786\": rpc error: code = NotFound desc = could not find container \"89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786\": container with ID starting with 89b2abd0ed1331b9c8e9bcff9ac2c0d51637a425d6ca81373cb4d5081ca20786 not found: ID does not exist" Nov 01 20:30:56.051300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.051166 5633 scope.go:122] "RemoveContainer" containerID="c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955" Nov 01 20:30:56.051580 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:30:56.051547 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955\": container with ID starting with c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955 not found: ID does not exist" containerID="c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955" Nov 01 20:30:56.051580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.051569 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955"} err="failed to get container status \"c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955\": rpc error: code = NotFound desc = could not find container \"c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955\": container with ID starting with c71b4d955fd51090a4d3603e6a3156d9a929f60202417e59539a297c1b064955 not found: ID does not exist" Nov 01 20:30:56.234021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.233988 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s"] Nov 01 20:30:56.237783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.237756 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mlh7s"] Nov 01 20:30:56.342665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.342631 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2"] Nov 01 20:30:56.346532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.346508 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-rv4r2"] Nov 01 20:30:56.453113 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.453076 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v"] Nov 01 20:30:56.459567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.459540 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-5s25v"] Nov 01 20:30:56.468964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.468934 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" path="/var/lib/kubelet/pods/08ccf9e0-4a1f-44e9-b032-13d7d763dbdd/volumes" Nov 01 20:30:56.469586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.469564 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" path="/var/lib/kubelet/pods/1ed39b20-1f38-4a11-a398-ccb791e0a159/volumes" Nov 01 20:30:56.470129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:56.470110 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" path="/var/lib/kubelet/pods/437e5a55-57f6-4a8b-a2b0-893259e41ad3/volumes" Nov 01 20:30:59.787007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:30:59.786965 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:03.937136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:03.937108 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:31:03.937638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:03.937176 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:31:04.568832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:04.568795 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:07.782047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:07.782005 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:31:09.572604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:09.572561 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:14.046142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:14.046111 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz"] Nov 01 20:31:14.094499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:14.094462 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ebfb90-6hhpz"] Nov 01 20:31:14.241347 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:14.241305 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:14.467708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:14.467678 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab51c6f-cf5e-4397-805f-70d307fc6105" path="/var/lib/kubelet/pods/8ab51c6f-cf5e-4397-805f-70d307fc6105/volumes" Nov 01 20:31:14.476043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:14.476009 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:16.157692 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:16.157656 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:31:16.332980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:16.332950 5633 scope.go:122] "RemoveContainer" containerID="8c382f79ae0b3e55d05c98ae92742e22f4ea6633feabe9df21d2e74962d5ea55" Nov 01 20:31:16.343945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:16.343922 5633 scope.go:122] "RemoveContainer" containerID="c31da941ac225b9572c41ba4e89eb023478b1cd1365a0467b308878d44548ae5" Nov 01 20:31:16.354148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:16.354112 5633 scope.go:122] "RemoveContainer" containerID="ce09a11f5223c1193c543fd64c849607c73ce47331cec66fbb036898a848de5e" Nov 01 20:31:16.363880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:16.363860 5633 scope.go:122] "RemoveContainer" containerID="34e7b34dcb67c0a225678c582b455c399a5bf08161bdafb241b90849314cedcf" Nov 01 20:31:16.374168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:16.374147 5633 scope.go:122] "RemoveContainer" containerID="cf0b89d5f717d3fd7ac2268e8bc234ae0cc390933a98c1848ddb37bba92ffff6" Nov 01 20:31:16.384389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:16.384368 5633 scope.go:122] "RemoveContainer" containerID="44854443a1957b5a5b67dd3a08dd1cfb6453152c73c39f405a1bee1f414552d7" Nov 01 20:31:18.288689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.288651 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:18.756156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756126 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944"] Nov 01 20:31:18.756853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756833 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" Nov 01 20:31:18.756853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756852 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" Nov 01 20:31:18.757018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756879 5633 cpu_manager.go:414] "containerMap: removing container" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" Nov 01 20:31:18.757018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756884 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" Nov 01 20:31:18.757018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756900 5633 cpu_manager.go:414] "containerMap: removing container" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" Nov 01 20:31:18.757018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756905 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" Nov 01 20:31:18.757018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756921 5633 cpu_manager.go:414] "containerMap: removing container" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" Nov 01 20:31:18.757018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.756925 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" Nov 01 20:31:18.757018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.757015 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed39b20-1f38-4a11-a398-ccb791e0a159" containerName="istio-proxy" Nov 01 20:31:18.757234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.757028 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" Nov 01 20:31:18.757234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.757035 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="437e5a55-57f6-4a8b-a2b0-893259e41ad3" containerName="istio-proxy" Nov 01 20:31:18.757234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.757042 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ccf9e0-4a1f-44e9-b032-13d7d763dbdd" containerName="istio-proxy" Nov 01 20:31:18.763486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.763458 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.869907 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.869860 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944"] Nov 01 20:31:18.877369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877311 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.877575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877497 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411d425-6b52-467d-819f-af21d34e68b6-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.877575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877556 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.877741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877724 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz79h\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-kube-api-access-nz79h\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.877809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877779 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.877873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877804 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.877873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877832 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.878009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.877984 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411d425-6b52-467d-819f-af21d34e68b6-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.878063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.878043 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.980845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.980787 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411d425-6b52-467d-819f-af21d34e68b6-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.980858 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.980971 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.981066 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411d425-6b52-467d-819f-af21d34e68b6-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.981129 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.981219 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz79h\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-kube-api-access-nz79h\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.981264 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.981293 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.981402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.981320 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.984013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.983971 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.984806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.984626 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411d425-6b52-467d-819f-af21d34e68b6-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.985058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.985036 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.988457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.985882 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.988457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.986144 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.991800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.991767 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411d425-6b52-467d-819f-af21d34e68b6-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:18.992096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:18.992073 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:19.038611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.038528 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:19.038792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.038620 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz79h\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-kube-api-access-nz79h\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-tc944\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:19.078758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.078719 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:19.539723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.539688 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:19.561445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.561408 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944"] Nov 01 20:31:19.597541 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:31:19.597508 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411d425_6b52_467d_819f_af21d34e68b6.slice/crio-f711c6325fd68f138303f93641fe3cab5f725b14f2691ffa34e6edd7eaf54551 WatchSource:0}: Error finding container f711c6325fd68f138303f93641fe3cab5f725b14f2691ffa34e6edd7eaf54551: Status 404 returned error can't find the container with id f711c6325fd68f138303f93641fe3cab5f725b14f2691ffa34e6edd7eaf54551 Nov 01 20:31:19.600091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.600060 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:19.600155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.600130 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:19.600196 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.600158 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:19.983209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:19.983176 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:20.202278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:20.202238 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" event={"ID":"9411d425-6b52-467d-819f-af21d34e68b6","Type":"ContainerStarted","Data":"9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c"} Nov 01 20:31:20.202590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:20.202564 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" event={"ID":"9411d425-6b52-467d-819f-af21d34e68b6","Type":"ContainerStarted","Data":"f711c6325fd68f138303f93641fe3cab5f725b14f2691ffa34e6edd7eaf54551"} Nov 01 20:31:21.079434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:21.079398 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:21.081176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:21.081152 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-tc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 20:31:21.081295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:21.081209 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 20:31:21.815380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:21.812832 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:21.937922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:21.937889 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:22.079692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:22.079607 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-tc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 20:31:22.079692 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:22.079664 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 20:31:23.080023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:23.079983 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-tc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 20:31:23.080630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:23.080068 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 20:31:23.320618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:23.320581 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:23.776901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:23.776866 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:23.812091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:23.812027 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podStartSLOduration=5.812011004 podStartE2EDuration="5.812011004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:20.343240828 +0000 UTC m=+1688.017500084" watchObservedRunningTime="2026-11-01 20:31:23.812011004 +0000 UTC m=+1691.486270260" Nov 01 20:31:23.814853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:23.814828 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8"] Nov 01 20:31:23.815109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:23.815087 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" containerID="cri-o://8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f" gracePeriod=30 Nov 01 20:31:24.079732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.079661 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-tc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 20:31:24.080084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.079922 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 20:31:24.348292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.348189 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:24.859613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.859582 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d"] Nov 01 20:31:24.864052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.864025 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929122 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istiod-ca-cert\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929168 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-envoy\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929193 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-socket\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929217 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-certs\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929236 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxh5f\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-kube-api-access-hxh5f\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929256 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-token\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929285 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-credential-socket\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929308 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-data\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.930011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.929416 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-podinfo\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:24.931292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.931200 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:24.933751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:24.933725 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d"] Nov 01 20:31:25.012952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.003775 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f154f7-openshift-default-dockercfg-82gjq\"" Nov 01 20:31:25.030689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.030654 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-socket\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.030689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.030698 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-certs\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.030962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.030719 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxh5f\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-kube-api-access-hxh5f\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.030962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.030745 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-token\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.030962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.030791 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-credential-socket\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.030962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.030818 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-data\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.030962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.030905 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-podinfo\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.031257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.031076 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istiod-ca-cert\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.031257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.031107 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-socket\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.031257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.031135 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-envoy\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.031447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.031318 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-credential-socket\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.031447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.031392 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-data\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.031678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.031657 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-certs\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.031766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.031747 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istiod-ca-cert\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.034149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.034121 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-podinfo\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.034279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.034156 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-envoy\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.052975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.052929 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:25.079268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.079223 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-tc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 20:31:25.079484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.079297 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 20:31:25.111301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.111203 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxh5f\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-kube-api-access-hxh5f\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.115242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.115173 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-token\") pod \"e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.187881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.186604 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:25.223653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.223472 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:25.350407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.350299 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:25.743482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.743446 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d"] Nov 01 20:31:25.743683 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:31:25.743594 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24418ce_cbc8_414a_92f7_9f2ed7f338b3.slice/crio-feab7b2587b3e86068267b72ee3bf23e54fd8aa60ff1c3e31da07fe30f021238 WatchSource:0}: Error finding container feab7b2587b3e86068267b72ee3bf23e54fd8aa60ff1c3e31da07fe30f021238: Status 404 returned error can't find the container with id feab7b2587b3e86068267b72ee3bf23e54fd8aa60ff1c3e31da07fe30f021238 Nov 01 20:31:25.746321 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.746287 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:25.746487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.746422 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:25.746487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:25.746473 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:26.079592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:26.079494 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-tc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 20:31:26.079592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:26.079547 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 20:31:26.319619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:26.319569 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" event={"ID":"d24418ce-cbc8-414a-92f7-9f2ed7f338b3","Type":"ContainerStarted","Data":"f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735"} Nov 01 20:31:26.320117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:26.319643 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" event={"ID":"d24418ce-cbc8-414a-92f7-9f2ed7f338b3","Type":"ContainerStarted","Data":"feab7b2587b3e86068267b72ee3bf23e54fd8aa60ff1c3e31da07fe30f021238"} Nov 01 20:31:26.437941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:26.437821 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podStartSLOduration=2.437804442 podStartE2EDuration="2.437804442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:26.434735899 +0000 UTC m=+1694.108995159" watchObservedRunningTime="2026-11-01 20:31:26.437804442 +0000 UTC m=+1694.112063893" Nov 01 20:31:26.971736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:26.971698 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:27.084611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.084579 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:27.085050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.085025 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:27.085934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.085907 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:31:27.093949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.093918 5633 patch_prober.go:29] interesting pod/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:31:27.094069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.093964 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:31:27.188072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.188038 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:27.194245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.194219 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:27.333752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.333646 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:27.335576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:27.335546 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:31:29.045853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.045815 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2"] Nov 01 20:31:29.090838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.090758 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:31:29.094813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.094784 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e28c32c-dpfq2"] Nov 01 20:31:29.157154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.157122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:29.201533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201491 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-socket\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.201533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201539 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-kube-api-access-8tl6m\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-kube-api-access-8tl6m\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.201804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201686 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-envoy\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.201804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201785 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-data\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.201920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201822 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-certs\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.201920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201865 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-credential-socket\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.202024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201929 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-token\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-token\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.202024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201982 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-podinfo\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.202024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201817 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-socket" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:29.202024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.201991 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-data" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:29.203499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.202256 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-certs" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:29.203499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.202512 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istiod-ca-cert" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:29.203623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.203602 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-credential-socket" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:29.203682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.202032 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istiod-ca-cert\") pod \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\" (UID: \"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f\") " Nov 01 20:31:29.204858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204294 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.204858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204318 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.204858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204376 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.204858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204391 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.204858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204415 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.204858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204715 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-kube-api-access-8tl6m" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "kube-api-access-8tl6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:29.204858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204802 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-envoy" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:29.205298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.204880 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-token" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:29.205298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.205147 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-podinfo" pod "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" (UID: "c5f2ae96-3051-4987-84f5-1ee9c3a1b48f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:31:29.306311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.306274 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.306311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.306306 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.306311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.306318 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tl6m\" (UniqueName: \"kubernetes.io/projected/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-kube-api-access-8tl6m\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.306682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.306371 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:29.339879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.339836 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:29.354211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.354122 5633 generic.go:356] "Generic (PLEG): container finished" podID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerID="8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f" exitCode=0 Nov 01 20:31:29.354211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.354167 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" event={"ID":"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f","Type":"ContainerDied","Data":"8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f"} Nov 01 20:31:29.354521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.354235 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" event={"ID":"c5f2ae96-3051-4987-84f5-1ee9c3a1b48f","Type":"ContainerDied","Data":"9ec6aa2042b697d357e2fcd4c0ae07df23bed9d40bda526c90d8b340adade774"} Nov 01 20:31:29.354521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.354176 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8" Nov 01 20:31:29.354521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.354268 5633 scope.go:122] "RemoveContainer" containerID="8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f" Nov 01 20:31:29.366931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.366912 5633 scope.go:122] "RemoveContainer" containerID="d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b" Nov 01 20:31:29.377278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.377253 5633 scope.go:122] "RemoveContainer" containerID="8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f" Nov 01 20:31:29.377650 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:29.377628 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f\": container with ID starting with 8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f not found: ID does not exist" containerID="8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f" Nov 01 20:31:29.377736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.377661 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f"} err="failed to get container status \"8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f\": rpc error: code = NotFound desc = could not find container \"8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f\": container with ID starting with 8e92c704b9bf956a331423039440abbcead11f915e9c9a2b661dd8848f33138f not found: ID does not exist" Nov 01 20:31:29.377736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.377680 5633 scope.go:122] "RemoveContainer" containerID="d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b" Nov 01 20:31:29.377988 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:29.377971 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b\": container with ID starting with d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b not found: ID does not exist" containerID="d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b" Nov 01 20:31:29.378060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.377995 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b"} err="failed to get container status \"d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b\": rpc error: code = NotFound desc = could not find container \"d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b\": container with ID starting with d199bd061ef9f765d65236072e8906193afeb7a3e90ade2a4dc2c9e98c54837b not found: ID does not exist" Nov 01 20:31:29.478825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.478780 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:29.519915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.519882 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8"] Nov 01 20:31:29.523857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:29.523827 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-202d23-openshift-default-77cbc5f677-w7kx8"] Nov 01 20:31:30.044996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:30.044955 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct"] Nov 01 20:31:30.048438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:30.048409 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d05e489c-sqhct"] Nov 01 20:31:30.347533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:30.347448 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:30.468048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:30.468013 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff429fe-cecd-4ab4-8155-de32c795f4fc" path="/var/lib/kubelet/pods/0ff429fe-cecd-4ab4-8155-de32c795f4fc/volumes" Nov 01 20:31:30.468543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:30.468527 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4415b8c-2489-4872-a47b-56ffbaa1a212" path="/var/lib/kubelet/pods/a4415b8c-2489-4872-a47b-56ffbaa1a212/volumes" Nov 01 20:31:30.468964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:30.468952 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" path="/var/lib/kubelet/pods/c5f2ae96-3051-4987-84f5-1ee9c3a1b48f/volumes" Nov 01 20:31:43.704141 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:43.704100 5633 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-78.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:31:44.524178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.524140 5633 generic.go:356] "Generic (PLEG): container finished" podID="ae182c81-580e-4e54-bf4b-9696750c18ed" containerID="1bb86b7026a6ac6ede0f2f938c1a62663f43d5759e8430d3abe1e0270654219d" exitCode=1 Nov 01 20:31:44.524456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.524175 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" event={"ID":"ae182c81-580e-4e54-bf4b-9696750c18ed","Type":"ContainerDied","Data":"1bb86b7026a6ac6ede0f2f938c1a62663f43d5759e8430d3abe1e0270654219d"} Nov 01 20:31:44.524673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.524655 5633 scope.go:122] "RemoveContainer" containerID="1bb86b7026a6ac6ede0f2f938c1a62663f43d5759e8430d3abe1e0270654219d" Nov 01 20:31:44.525174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.525157 5633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:31:44.526018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.525982 5633 generic.go:356] "Generic (PLEG): container finished" podID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" exitCode=1 Nov 01 20:31:44.526125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.526032 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" event={"ID":"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864","Type":"ContainerDied","Data":"4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e"} Nov 01 20:31:44.526542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.526464 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 20:31:44.527803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.527782 5633 generic.go:356] "Generic (PLEG): container finished" podID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerID="982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891" exitCode=0 Nov 01 20:31:44.527882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.527858 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" event={"ID":"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4","Type":"ContainerDied","Data":"982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891"} Nov 01 20:31:44.528259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.528244 5633 scope.go:122] "RemoveContainer" containerID="982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891" Nov 01 20:31:44.529747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.529725 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:44.529822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.529790 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:44.529860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.529820 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:31:44.886590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:44.886490 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal" podUID="c04d0eff16222e7baef223e711863fea" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:31:45.551131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.551054 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" event={"ID":"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4","Type":"ContainerStarted","Data":"be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9"} Nov 01 20:31:45.553377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.553353 5633 generic.go:356] "Generic (PLEG): container finished" podID="2e038c41-6c0d-45d8-8fcd-2c8ac51ba295" containerID="5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf" exitCode=1 Nov 01 20:31:45.553506 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.553422 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" event={"ID":"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295","Type":"ContainerDied","Data":"5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf"} Nov 01 20:31:45.553873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.553853 5633 scope.go:122] "RemoveContainer" containerID="5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf" Nov 01 20:31:45.555806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.555784 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" event={"ID":"ae182c81-580e-4e54-bf4b-9696750c18ed","Type":"ContainerStarted","Data":"9c7d7f6e96af3929fb91bb62d7ead1495f3197d8681a17481fa216ed52a1dba9"} Nov 01 20:31:45.556000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.555983 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:31:45.723368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.723314 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:31:45.857813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.857725 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:31:45.859272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.859246 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-c259f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 20:31:45.859432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:45.859295 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 20:31:46.157400 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:46.157293 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:31:46.580855 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:46.580822 5633 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod maas-controller-645c58c6f7-mc6zb_opendatahub(f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError" Nov 01 20:31:46.581035 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:46.580862 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" Nov 01 20:31:46.858311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:46.858219 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-c259f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 20:31:46.858311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:46.858291 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 20:31:47.462940 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:47.462900 5633 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod ai-gateway-controller-54d6db5d4f-6rv48_opendatahub(2e038c41-6c0d-45d8-8fcd-2c8ac51ba295): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError" Nov 01 20:31:47.462940 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:47.462945 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" podUID="2e038c41-6c0d-45d8-8fcd-2c8ac51ba295" Nov 01 20:31:47.858368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:47.858246 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-c259f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 20:31:47.858368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:47.858314 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 20:31:48.733978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:48.733945 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:31:48.734505 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:48.734489 5633 scope.go:122] "RemoveContainer" containerID="5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf" Nov 01 20:31:48.734829 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:48.734801 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=ai-gateway-controller-54d6db5d4f-6rv48_opendatahub(2e038c41-6c0d-45d8-8fcd-2c8ac51ba295)\"" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" podUID="2e038c41-6c0d-45d8-8fcd-2c8ac51ba295" Nov 01 20:31:48.857758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:48.857725 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-c259f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 20:31:48.857950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:48.857791 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 20:31:48.937902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:48.937871 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:31:48.938294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:48.938280 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 20:31:48.938595 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:48.938575 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=maas-controller-645c58c6f7-mc6zb_opendatahub(f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864)\"" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" Nov 01 20:31:49.858183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:49.858135 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-c259f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 20:31:49.858653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:49.858209 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 20:31:50.857528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:50.857490 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-c259f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 01 20:31:50.857734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:50.857559 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 01 20:31:51.862323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:51.862293 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:31:51.863628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:51.863605 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:31:53.704958 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:53.704918 5633 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-78.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:31:55.605445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:55.605410 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:56.326500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:56.326462 5633 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:31:56.326996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:56.326971 5633 scope.go:122] "RemoveContainer" containerID="5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf" Nov 01 20:31:56.459662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:56.459629 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 20:31:56.559563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:31:56.559521 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:31:58.108193 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:58.108146 5633 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod ai-gateway-controller-54d6db5d4f-6rv48_opendatahub(2e038c41-6c0d-45d8-8fcd-2c8ac51ba295): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError" Nov 01 20:31:58.108635 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:58.108201 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" podUID="2e038c41-6c0d-45d8-8fcd-2c8ac51ba295" Nov 01 20:31:58.273499 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:58.273459 5633 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod maas-controller-645c58c6f7-mc6zb_opendatahub(f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError" Nov 01 20:31:58.273694 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:31:58.273515 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" Nov 01 20:32:00.706401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:00.706363 5633 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:32:00.706817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:00.706791 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 20:32:00.707113 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:00.707091 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=maas-controller-645c58c6f7-mc6zb_opendatahub(f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864)\"" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" Nov 01 20:32:01.992068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:01.992034 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:02.126953 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:02.126907 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:03.705641 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:03.705597 5633 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-78.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:32:04.459746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:04.459706 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 20:32:06.320768 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:06.320718 5633 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod maas-controller-645c58c6f7-mc6zb_opendatahub(f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError" Nov 01 20:32:06.320768 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:06.320774 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" Nov 01 20:32:08.549861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:08.549828 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:11.460273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:11.460244 5633 scope.go:122] "RemoveContainer" containerID="5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf" Nov 01 20:32:13.254653 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:13.254615 5633 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod ai-gateway-controller-54d6db5d4f-6rv48_opendatahub(2e038c41-6c0d-45d8-8fcd-2c8ac51ba295): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError" Nov 01 20:32:13.255072 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:13.254664 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" podUID="2e038c41-6c0d-45d8-8fcd-2c8ac51ba295" Nov 01 20:32:13.705909 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:13.705787 5633 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-78.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:32:15.129325 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:15.129276 5633 kubelet_node_status.go:524] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-11-01T20:32:05Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-11-01T20:32:05Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-11-01T20:32:05Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-11-01T20:32:05Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-128-78.us-west-1.compute.internal\": Patch \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/nodes/ip-10-0-128-78.us-west-1.compute.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:32:15.554792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:15.554750 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:16.170192 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:16.170146 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:32:16.507297 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:16.507256 5633 scope.go:122] "RemoveContainer" containerID="4df4ef295fbd434aadfb03f65ae0ec423f94883b8dc74a6c49b6f1f99b6af6dd" Nov 01 20:32:16.530852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:16.530476 5633 scope.go:122] "RemoveContainer" containerID="d7b81a76468d69e29074a215b0dd8598d962bccdfce44188ff202572f27f0d0a" Nov 01 20:32:16.550999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:16.550562 5633 scope.go:122] "RemoveContainer" containerID="5fe6e40cd9a61e807876affba0801cce9a33a00d698a801aad2e746a96441fb2" Nov 01 20:32:16.571390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:16.571307 5633 scope.go:122] "RemoveContainer" containerID="881b4285f2258fad151b01c5277313f5d934c40a6498449e171b0a7544af72d9" Nov 01 20:32:16.571865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:16.571846 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4dzk7" Nov 01 20:32:16.588915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:16.588891 5633 scope.go:122] "RemoveContainer" containerID="b28559eeb41cb2e407df29502921e6aefcff8c1a246b6ce3583fa98794d7c346" Nov 01 20:32:17.213998 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.213966 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:17.414311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.414264 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:17.626883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.626779 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:17.696402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696356 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696469 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6a675a\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696363 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696496 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696515 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696529 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696552 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z5b7z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696505 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696637 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-ait-393dc263-openshift-default-dockercfg-ssfv2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696356 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696717 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696735 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696743 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.696931 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696773 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dt6v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696971 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.696971 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697053 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697055 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-wskl6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697063 5633 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697095 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697102 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697107 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-knqbm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697133 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6a675a-openshift-default-dockercfg-85chn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697151 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697225 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697155 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697236 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697199 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697213 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697239 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697161 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697285 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-71335882\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697295 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697309 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697268 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697343 5633 trace.go:236] Trace[294696715]: "Reflector WatchList" name:object-"opendatahub"/"maas-parameters" (01-Nov-2026 20:31:45.580) (total time: 32117ms): Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: Trace[294696715]: [32.117090393s] [32.117090393s] END Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697352 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697357 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697359 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697353 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697390 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-6vk7hqa1puoju\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697391 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697421 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697390 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2cc24d-openshift-default-dockercfg-b4lxl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697443 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697464 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"default-dockercfg-qdjvd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697474 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1b1134-openshift-default-dockercfg-pvzt4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697490 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697498 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697521 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"distinct-w3-1b1134-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697542 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-dockercfg-877r8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697551 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697562 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697598 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697596 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697621 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697623 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697594 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697643 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.697636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697653 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697670 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-71335882-dockercfg-p9gc4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697680 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bqhkg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697706 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697745 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697774 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697787 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5ch2z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697821 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697836 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7tn6l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697848 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l79xf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697822 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697873 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"service-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697885 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697916 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697919 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697916 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f154f7-openshift-default-dockercfg-82gjq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697937 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697940 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697953 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697958 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697965 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697971 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.697845 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698021 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698029 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jh6hf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698035 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ljq7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698030 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698051 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698027 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698071 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ck4qq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698073 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698082 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698079 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698101 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698105 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698105 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698121 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-w2kmv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698115 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698113 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-df41b8-openshift-default-dockercfg-75q9g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698143 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698143 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698156 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698153 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c9o03dg9n84k5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698165 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698175 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698104 5633 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698177 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698197 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698202 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-f5888\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698205 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698203 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698167 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sx9cg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698230 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698234 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8nknn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698245 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"distinct-2-w0-d204f4-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698258 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698265 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698270 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698261 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698272 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-2cc24d\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698271 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-99d504-openshift-default-dockercfg-55qzx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698289 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698289 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-df41b8-dockercfg-mt969\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698299 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698317 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698316 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-mt-71335882-openshift-default-dockercfg-f4t24\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698324 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-2cc24d-dockercfg-zwmf9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698370 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698381 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698419 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698436 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698445 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698447 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698461 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698469 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-oauth-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698476 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698452 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698493 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698492 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-393dc263\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698493 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"default-dockercfg-t4w2r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698516 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698505 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698532 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698542 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698545 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698548 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698514 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-393dc263-dockercfg-8xzw9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698561 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698512 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698562 5633 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698579 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698574 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698592 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-76lth\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698598 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698604 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"distinct-2-w3-1b1134-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698611 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698622 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-hh5t6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698625 5633 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698622 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698638 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698644 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698649 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-df41b8\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698649 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698664 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698659 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698673 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698670 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sbflc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698684 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698688 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698695 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-db7jz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698666 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698704 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698713 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698729 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698732 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698706 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698745 5633 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698751 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698759 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698764 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698769 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ft87h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698754 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698780 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2r24h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698727 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4jhrn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698797 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698803 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-45cmh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698771 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698826 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698831 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698805 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698842 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:17.698706 5633 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/opendatahub/events/kserve-controller-manager-776cc66d47-4dzk7.18e404b09a03cb8d\": http2: client connection lost" event="&Event{ObjectMeta:{kserve-controller-manager-776cc66d47-4dzk7.18e404b09a03cb8d opendatahub 17083 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:kserve-controller-manager-776cc66d47-4dzk7,UID:ae182c81-580e-4e54-bf4b-9696750c18ed,APIVersion:v1,ResourceVersion:17015,FieldPath:spec.containers{manager},},Reason:Pulling,Message:Pulling image \"quay.io/opendatahub/kserve-controller@sha256:6e72a4ac896e998acf222204e10868106932adfbfd0dafdc0a9c1e4d58d0627d\",Source:EventSource{Component:kubelet,Host:ip-10-0-128-78.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:08:28 +0000 UTC,LastTimestamp:2026-11-01 20:31:44.525823965 +0000 UTC m=+1712.200083199,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-78.us-west-1.compute.internal,}" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698860 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698864 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698867 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698783 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:17.698855 5633 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-78.us-west-1.compute.internal?timeout=10s\": http2: client connection lost" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698888 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698893 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698858 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698901 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698902 5633 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698838 5633 trace.go:236] Trace[580091734]: "Reflector WatchList" name:object-"opendatahub"/"ai-gateway-controller-parameters" (01-Nov-2026 20:31:46.462) (total time: 31236ms): Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: Trace[580091734]: [31.23635014s] [31.23635014s] END Nov 01 20:32:17.699372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.698890 5633 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Nov 01 20:32:17.767184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.767132 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:17.857367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.857323 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 20:32:17.857578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:17.857378 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 20:32:18.460107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.460074 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 20:32:18.460583 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:18.460439 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=maas-controller-645c58c6f7-mc6zb_opendatahub(f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864)\"" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" Nov 01 20:32:18.566397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.566361 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 20:32:18.566397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.566360 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 20:32:18.570481 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.570454 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:32:18.616023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.615987 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 20:32:18.620405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.620369 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:32:18.621137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.621111 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 20:32:18.621304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.621113 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 20:32:18.665410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.665370 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:32:18.665410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.665371 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.665629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.665457 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:32:18.665629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.665471 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 20:32:18.665629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.665479 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.719976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.719938 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.720205 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.720121 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.764320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764283 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.764320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764361 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:32:18.764632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764383 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:32:18.764632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764585 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 20:32:18.764632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764628 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:32:18.764777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764636 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 20:32:18.764777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764643 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 20:32:18.764777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.764711 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-df41b8-openshift-default-dockercfg-75q9g\"" Nov 01 20:32:18.769921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.769891 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 20:32:18.769921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.769892 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 20:32:18.770148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.769960 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\"" Nov 01 20:32:18.770148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.769970 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-393dc263-dockercfg-8xzw9\"" Nov 01 20:32:18.770636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.770615 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:32:18.770769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.770614 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.770769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.770622 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 20:32:18.770965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.770665 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.813964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.813932 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 20:32:18.814140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.814119 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dt6v\"" Nov 01 20:32:18.814238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.814127 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4jhrn\"" Nov 01 20:32:18.814238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.814199 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5ch2z\"" Nov 01 20:32:18.820859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.820832 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.821028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.820867 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"distinct-w3-1b1134-kserve-self-signed-certs\"" Nov 01 20:32:18.821028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.820871 5633 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:32:18.821206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.820924 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 20:32:18.821206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821075 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"default-dockercfg-qdjvd\"" Nov 01 20:32:18.821206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821109 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 20:32:18.821206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821192 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 20:32:18.821391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821273 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 20:32:18.821658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821642 5633 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:32:18.821789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821771 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-db7jz\"" Nov 01 20:32:18.821919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821808 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.821919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821815 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\"" Nov 01 20:32:18.822046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.821972 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-hh5t6\"" Nov 01 20:32:18.822046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822012 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 20:32:18.822152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822072 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.822152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822075 5633 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:32:18.822152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822105 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.822152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822141 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sx9cg\"" Nov 01 20:32:18.822393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822188 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 20:32:18.822393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822230 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 20:32:18.822485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822414 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.822485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822458 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:32:18.822551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822492 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:32:18.822551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822510 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:32:18.822551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.822533 5633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-78.us-west-1.compute.internal" event="NodeReady" Nov 01 20:32:18.848677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.848640 5633 generic.go:356] "Generic (PLEG): container finished" podID="b68f6371-dc7c-440c-8fda-13b8388e7221" containerID="61a13e941f54f597b6a1224ab83b5b6d3ef8806b5d77eade2950ad72a317b5a4" exitCode=1 Nov 01 20:32:18.848852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.848678 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" event={"ID":"b68f6371-dc7c-440c-8fda-13b8388e7221","Type":"ContainerDied","Data":"61a13e941f54f597b6a1224ab83b5b6d3ef8806b5d77eade2950ad72a317b5a4"} Nov 01 20:32:18.849100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.849083 5633 scope.go:122] "RemoveContainer" containerID="61a13e941f54f597b6a1224ab83b5b6d3ef8806b5d77eade2950ad72a317b5a4" Nov 01 20:32:18.863781 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863755 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863807 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863830 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863891 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863905 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-2cc24d-dockercfg-zwmf9\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863913 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863933 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863955 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863967 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-71335882-dockercfg-p9gc4\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.863991 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.864040 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6vk7hqa1puoju\"" Nov 01 20:32:18.864094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.864048 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 20:32:18.864639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.864123 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-df41b8\"" Nov 01 20:32:18.864639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.864208 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 20:32:18.864639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.864377 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.864958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.864937 5633 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:32:18.871446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.871420 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\"" Nov 01 20:32:18.871577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.871430 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 20:32:18.871577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.871452 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\"" Nov 01 20:32:18.871577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.871496 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\"" Nov 01 20:32:18.871577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.871567 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" Nov 01 20:32:18.871775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.871672 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 20:32:18.882743 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.882720 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.882871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.882746 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 20:32:18.885488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.885468 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-71335882\"" Nov 01 20:32:18.885488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.885476 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sbflc\"" Nov 01 20:32:18.888235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.888043 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 20:32:18.893789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.893750 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 20:32:18.893974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.893956 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 20:32:18.908252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.908227 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:32:18.911540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.911509 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 20:32:18.912718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.912695 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 20:32:18.914082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.914064 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 20:32:18.919971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.919948 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wskl6\"" Nov 01 20:32:18.921586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.921561 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 20:32:18.925402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.925379 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 20:32:18.925504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.925422 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 20:32:18.929832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.929802 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\"" Nov 01 20:32:18.929967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.929940 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-45cmh\"" Nov 01 20:32:18.930664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.930650 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.932006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.931989 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-f5888\"" Nov 01 20:32:18.934495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.934476 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.939956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.939882 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71335882-openshift-default-dockercfg-f4t24\"" Nov 01 20:32:18.940442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.940381 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.942382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.940560 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.942997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.942974 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"distinct-2-w0-d204f4-kserve-self-signed-certs\"" Nov 01 20:32:18.950397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.948634 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 20:32:18.953652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.953630 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c9o03dg9n84k5\"" Nov 01 20:32:18.965052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.965014 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jh6hf\"" Nov 01 20:32:18.965800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.965776 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.966847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.966826 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2r24h\"" Nov 01 20:32:18.968926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.968910 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.975833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.975768 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.985229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.985206 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-393dc263-openshift-default-dockercfg-ssfv2\"" Nov 01 20:32:18.993062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.993012 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 20:32:18.993311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.993293 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"kube-root-ca.crt\"" Nov 01 20:32:18.993533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:18.993414 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7tn6l\"" Nov 01 20:32:19.002343 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.002300 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 20:32:19.002554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.002392 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 20:32:19.003623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.003600 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ljq7\"" Nov 01 20:32:19.009847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.009503 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-393dc263\"" Nov 01 20:32:19.009847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.009674 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6a675a\"" Nov 01 20:32:19.009847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.009712 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"distinct-2-w3-1b1134-kserve-self-signed-certs\"" Nov 01 20:32:19.010231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.009971 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 20:32:19.010901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.010861 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-877r8\"" Nov 01 20:32:19.011909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.011886 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.023615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.023578 5633 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:32:19.027321 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.027294 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 20:32:19.029659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.029608 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1b1134-openshift-default-dockercfg-pvzt4\"" Nov 01 20:32:19.032541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.032519 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:32:19.034605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.034584 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 20:32:19.034949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.034923 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7564a258-z94j9","odh-ai-gateway-infra/maas-api-revoke-sub-2b5d3d22-ckhjp","odh-ai-gateway-infra/maas-api-revoke-sub-737debe1-44m6l","odh-ai-gateway-infra/maas-api-revoke-sub-e49108ba-lzhk4"] Nov 01 20:32:19.035058 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.034991 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl","openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8"] Nov 01 20:32:19.035911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.035893 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" Nov 01 20:32:19.036041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.036027 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" Nov 01 20:32:19.038737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.038699 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" Nov 01 20:32:19.038854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.038843 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" Nov 01 20:32:19.039194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.039183 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" Nov 01 20:32:19.039274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.039265 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f2ae96-3051-4987-84f5-1ee9c3a1b48f" containerName="istio-proxy" Nov 01 20:32:19.043552 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.043521 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.049168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.049135 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 20:32:19.049430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.049408 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:32:19.049937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.049825 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.051179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.051160 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.052527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.052506 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-w2kmv\"" Nov 01 20:32:19.052812 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.052767 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2cc24d-openshift-default-dockercfg-b4lxl\"" Nov 01 20:32:19.056735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.056712 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-df41b8-dockercfg-mt969\"" Nov 01 20:32:19.062607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.062577 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.065231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.065206 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.078719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.078692 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.090240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.090200 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 20:32:19.091401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.091321 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 20:32:19.098352 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.092508 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.099256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.099225 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 20:32:19.109048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.108551 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"default-dockercfg-t4w2r\"" Nov 01 20:32:19.109048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.108679 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\"" Nov 01 20:32:19.110538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.110435 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 20:32:19.116893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.116867 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f154f7-openshift-default-dockercfg-82gjq\"" Nov 01 20:32:19.117874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.117852 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 20:32:19.119554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.119531 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 20:32:19.121011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.120991 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\"" Nov 01 20:32:19.121178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.121032 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:32:19.127348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.127303 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 20:32:19.142874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.142841 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 20:32:19.143061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.142892 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" Nov 01 20:32:19.143061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.142936 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-knqbm\"" Nov 01 20:32:19.143061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.142966 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\"" Nov 01 20:32:19.144929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.144907 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 20:32:19.149686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.149664 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 20:32:19.164521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.164494 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:32:19.167678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.167647 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.167868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.167846 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.167993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.167976 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f020231b-053e-410a-b714-dbc4baedf14b-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.168122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.168109 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.168283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.168268 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.168484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.168464 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.168621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.168606 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.168726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.168714 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.168833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.168820 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.168983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.168967 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.169141 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.169124 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jb9\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-kube-api-access-48jb9\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.169287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.169270 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.169463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.169446 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlwk\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-kube-api-access-pdlwk\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.169613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.169597 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.169751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.169737 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.169845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.169832 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.169965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.169951 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f020231b-053e-410a-b714-dbc4baedf14b-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.170137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.170123 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.171911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.171884 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.179204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.179175 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 20:32:19.179607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.179549 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z5b7z\"" Nov 01 20:32:19.184061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.184017 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 20:32:19.184190 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.184122 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.191791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.191767 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 20:32:19.191934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.191811 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:32:19.194631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.194609 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:32:19.196361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.196305 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 01 20:32:19.199322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.199287 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 20:32:19.205690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.205663 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\"" Nov 01 20:32:19.219272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.219237 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 20:32:19.219272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.219238 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ft87h\"" Nov 01 20:32:19.219489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.219380 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:32:19.223969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.223949 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.227929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.227847 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-2cc24d\"" Nov 01 20:32:19.235647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.235613 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.243028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.242998 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\"" Nov 01 20:32:19.255073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.255041 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-99d504-openshift-default-dockercfg-55qzx\"" Nov 01 20:32:19.255255 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.255145 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-76lth\"" Nov 01 20:32:19.260166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.260135 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.260385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.260367 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ck4qq\"" Nov 01 20:32:19.262393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.262374 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 20:32:19.266949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.266928 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:32:19.271614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.271580 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.271732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.271686 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.271809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.271749 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.271809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.271792 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f020231b-053e-410a-b714-dbc4baedf14b-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.271915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.271826 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.271915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.271897 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.271964 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272055 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272071 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272083 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272126 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272182 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272219 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jb9\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-kube-api-access-48jb9\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272243 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272262 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlwk\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-kube-api-access-pdlwk\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272296 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272374 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272393 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272431 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f020231b-053e-410a-b714-dbc4baedf14b-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272440 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.272828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272717 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.272915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.272898 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.273496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.273468 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f020231b-053e-410a-b714-dbc4baedf14b-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.274111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.274081 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.274359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.274320 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.274558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.274537 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l79xf\"" Nov 01 20:32:19.274625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.274566 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.274811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.274790 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.274891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.274839 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.280872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.275988 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.280872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.278493 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.280872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.280669 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 20:32:19.281163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.280883 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 20:32:19.281163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.281018 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.285107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.284859 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f020231b-053e-410a-b714-dbc4baedf14b-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.300355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.297520 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 20:32:19.300355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.297618 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.300355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.297700 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.300355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.297779 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.300355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.299832 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 20:32:19.303289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.303257 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" Nov 01 20:32:19.315495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.315462 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\"" Nov 01 20:32:19.316504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.316414 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8nknn\"" Nov 01 20:32:19.316664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.316532 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:32:19.317099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.317079 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:32:19.331663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.331625 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 20:32:19.331848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.331681 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bqhkg\"" Nov 01 20:32:19.331848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.331689 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6a675a-openshift-default-dockercfg-85chn\"" Nov 01 20:32:19.331848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.331746 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\"" Nov 01 20:32:19.335541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.335507 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:32:19.336355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.336105 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 20:32:19.338938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.338912 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.340803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.340767 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.343462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.343435 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jb9\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-kube-api-access-48jb9\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-hg8xl\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.344233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.344210 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlwk\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-kube-api-access-pdlwk\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-twxc8\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.348794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.348749 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\"" Nov 01 20:32:19.371259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.371227 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:19.382798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.382762 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:19.861908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.861804 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sqxc9" event={"ID":"b68f6371-dc7c-440c-8fda-13b8388e7221","Type":"ContainerStarted","Data":"d06d75a888f3dd128ffd2f21ac5d0e99357527b33fbb189b0f4484e74820a522"} Nov 01 20:32:19.891839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.891800 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl"] Nov 01 20:32:19.903736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.903696 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:19.903839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.903788 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:19.903839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.903833 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:19.904982 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.904949 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:19.905152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.905025 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:19.905152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.905064 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:19.906576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:19.906500 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8"] Nov 01 20:32:20.503069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.503034 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb6f177-e8fa-43ab-878f-4ba23da92b07" path="/var/lib/kubelet/pods/1eb6f177-e8fa-43ab-878f-4ba23da92b07/volumes" Nov 01 20:32:20.506198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.506146 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51bfff28-88c0-4407-9581-c17ff9494ae3" path="/var/lib/kubelet/pods/51bfff28-88c0-4407-9581-c17ff9494ae3/volumes" Nov 01 20:32:20.508129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.508082 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0fdec9-77cd-49d0-ad3e-8b3898705b0b" path="/var/lib/kubelet/pods/5f0fdec9-77cd-49d0-ad3e-8b3898705b0b/volumes" Nov 01 20:32:20.508778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.508749 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692a0ac6-6638-4b5f-b6ed-501ee613e59c" path="/var/lib/kubelet/pods/692a0ac6-6638-4b5f-b6ed-501ee613e59c/volumes" Nov 01 20:32:20.580725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.580202 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:20.892484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.889516 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" event={"ID":"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d","Type":"ContainerStarted","Data":"18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87"} Nov 01 20:32:20.892484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.889617 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" event={"ID":"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d","Type":"ContainerStarted","Data":"f00db3eb1756ccc6b23a6be0f75060c19a873c27da197f77083ea58cf5f5f53e"} Nov 01 20:32:20.893858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.893823 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" event={"ID":"f020231b-053e-410a-b714-dbc4baedf14b","Type":"ContainerStarted","Data":"13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0"} Nov 01 20:32:20.894077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:20.894062 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" event={"ID":"f020231b-053e-410a-b714-dbc4baedf14b","Type":"ContainerStarted","Data":"395bbbcbb939250718db6f55efe8cd2b33761dcae1ffbc1eb1c3e54719b55a38"} Nov 01 20:32:21.442760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:21.377928 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:21.442760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:21.442584 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:21.442760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:21.379869 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 20:32:21.442760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:21.442665 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 20:32:21.465631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:21.465595 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 20:32:21.466498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:21.466465 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 20:32:22.372958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:22.372922 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 20:32:22.373361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:22.372978 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 20:32:22.385370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:22.385311 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 20:32:22.407503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:22.407456 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 20:32:23.152722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:23.152685 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:23.371828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:23.371797 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 20:32:23.372045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:23.372024 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 20:32:23.383172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:23.383143 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 20:32:23.383672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:23.383198 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 20:32:23.671424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:23.671385 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:24.371968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:24.371848 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 20:32:24.371968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:24.371951 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 20:32:24.396568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:24.384497 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 20:32:24.396568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:24.384558 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 20:32:24.853223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:24.853179 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:25.373935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:25.372081 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 20:32:25.373935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:25.372137 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 20:32:25.383466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:25.383435 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 20:32:25.383621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:25.383486 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 20:32:26.372765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:26.372727 5633 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 20:32:26.373421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:26.373387 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 20:32:26.384357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:26.383222 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 01 20:32:26.384357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:26.383297 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 01 20:32:26.460098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:26.460065 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 20:32:27.062034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.062000 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p"] Nov 01 20:32:27.069750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.069716 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4717a66f-f698p"] Nov 01 20:32:27.342402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.342370 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:27.384204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.384113 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:27.389718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.387302 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:27.390962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.390935 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:32:27.399051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.398000 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:27.401158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.400975 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:27.405138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:27.405088 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:32:28.100146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.100105 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" event={"ID":"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864","Type":"ContainerStarted","Data":"64128474a9e8753156c095bfcf333a6079670019c7a0aacb166de84fb457ea95"} Nov 01 20:32:28.100971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.100948 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:32:28.190600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.190567 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx"] Nov 01 20:32:28.194891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.194873 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.261388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.261205 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx"] Nov 01 20:32:28.278510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.278471 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882b493-231f-4c6d-a84d-cf30e24a6056-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.278808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.278785 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.278926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.278912 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.279055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.279039 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz927\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-kube-api-access-jz927\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.279248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.279228 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.279518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.279497 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.279680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.279663 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.279857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.279838 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.285608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.285577 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.388970 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389017 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389063 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz927\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-kube-api-access-jz927\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389128 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389206 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389242 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389272 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389375 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389420 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.389520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.389471 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882b493-231f-4c6d-a84d-cf30e24a6056-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.391460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.390441 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.391460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.391283 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.392154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.392127 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.392362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.392164 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882b493-231f-4c6d-a84d-cf30e24a6056-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.408027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.407572 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.410233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.410192 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.447876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.447842 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.448127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.447960 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz927\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-kube-api-access-jz927\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-z2nvx\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.474060 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.474021 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c4c57d-4d34-4ac8-9194-a56e2cc5ccab" path="/var/lib/kubelet/pods/54c4c57d-4d34-4ac8-9194-a56e2cc5ccab/volumes" Nov 01 20:32:28.474655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.474574 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdbf44876-vjp4b"] Nov 01 20:32:28.475140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.475120 5633 scope.go:122] "RemoveContainer" containerID="5b22987e7aca19e749714fd881706c1d0ad8ecb4bfe4653e259043f8e53aacaf" Nov 01 20:32:28.510411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.510380 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:28.898872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.898844 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx"] Nov 01 20:32:28.901523 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:32:28.901491 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7882b493_231f_4c6d_a84d_cf30e24a6056.slice/crio-459b7460838218258dd23ae1c1cd82ec9c34e7083e3ec00268a98c2080503dce WatchSource:0}: Error finding container 459b7460838218258dd23ae1c1cd82ec9c34e7083e3ec00268a98c2080503dce: Status 404 returned error can't find the container with id 459b7460838218258dd23ae1c1cd82ec9c34e7083e3ec00268a98c2080503dce Nov 01 20:32:28.904476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.904436 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:28.904579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.904528 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:28.904579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.904573 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:28.942609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:28.942559 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podStartSLOduration=15.942540404 podStartE2EDuration="15.942540404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:28.937870746 +0000 UTC m=+1756.612130002" watchObservedRunningTime="2026-11-01 20:32:28.942540404 +0000 UTC m=+1756.616799658" Nov 01 20:32:29.113688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.113638 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" event={"ID":"7882b493-231f-4c6d-a84d-cf30e24a6056","Type":"ContainerStarted","Data":"88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16"} Nov 01 20:32:29.113891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.113709 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" event={"ID":"7882b493-231f-4c6d-a84d-cf30e24a6056","Type":"ContainerStarted","Data":"459b7460838218258dd23ae1c1cd82ec9c34e7083e3ec00268a98c2080503dce"} Nov 01 20:32:29.216482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.216439 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67848b78c7-ncqtf"] Nov 01 20:32:29.221064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.221034 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.292299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.292260 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67848b78c7-ncqtf"] Nov 01 20:32:29.417672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.417626 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-service-ca\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.418134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.417690 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb075952-c830-42ac-8c65-04d21a051342-console-oauth-config\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.418134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.417912 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb075952-c830-42ac-8c65-04d21a051342-console-serving-cert\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.418134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.417979 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zlmb\" (UniqueName: \"kubernetes.io/projected/eb075952-c830-42ac-8c65-04d21a051342-kube-api-access-4zlmb\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.418134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.418011 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb075952-c830-42ac-8c65-04d21a051342-tmp\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.418433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.418147 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-trusted-ca-bundle\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.418433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.418197 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-console-config\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.418433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.418243 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-oauth-serving-cert\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.501549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.501457 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:29.511141 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.511108 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:29.519606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.519570 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zlmb\" (UniqueName: \"kubernetes.io/projected/eb075952-c830-42ac-8c65-04d21a051342-kube-api-access-4zlmb\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.519778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.519620 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb075952-c830-42ac-8c65-04d21a051342-tmp\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.519778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.519720 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-trusted-ca-bundle\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.520032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.519749 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-console-config\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.520126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.520085 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-oauth-serving-cert\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.520126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.520101 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb075952-c830-42ac-8c65-04d21a051342-tmp\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.520261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.520180 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-service-ca\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.520261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.520213 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb075952-c830-42ac-8c65-04d21a051342-console-oauth-config\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.520535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.520453 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb075952-c830-42ac-8c65-04d21a051342-console-serving-cert\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.521271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.521230 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-trusted-ca-bundle\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.521408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.521274 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-service-ca\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.521408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.521358 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-console-config\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.521408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.521399 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb075952-c830-42ac-8c65-04d21a051342-oauth-serving-cert\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.523923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.523896 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb075952-c830-42ac-8c65-04d21a051342-console-oauth-config\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.524118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.524097 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb075952-c830-42ac-8c65-04d21a051342-console-serving-cert\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.594620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.594582 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zlmb\" (UniqueName: \"kubernetes.io/projected/eb075952-c830-42ac-8c65-04d21a051342-kube-api-access-4zlmb\") pod \"console-67848b78c7-ncqtf\" (UID: \"eb075952-c830-42ac-8c65-04d21a051342\") " pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.702512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.702447 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podStartSLOduration=56.702428388 podStartE2EDuration="56.702428388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:29.697758326 +0000 UTC m=+1757.372017588" watchObservedRunningTime="2026-11-01 20:32:29.702428388 +0000 UTC m=+1757.376687644" Nov 01 20:32:29.760628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.760519 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:29.837645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.837597 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:29.909845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.909773 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:29.950168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:29.949909 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podStartSLOduration=1.949890818 podStartE2EDuration="1.949890818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:29.948568482 +0000 UTC m=+1757.622827737" watchObservedRunningTime="2026-11-01 20:32:29.949890818 +0000 UTC m=+1757.624150077" Nov 01 20:32:30.125321 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:30.125230 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" event={"ID":"2e038c41-6c0d-45d8-8fcd-2c8ac51ba295","Type":"ContainerStarted","Data":"9da2e32dfae8c0c51a1506cb5bd985dcb45095d053450f9ad30b37c99585561f"} Nov 01 20:32:30.256238 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:30.256208 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67848b78c7-ncqtf"] Nov 01 20:32:30.256683 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:32:30.256656 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb075952_c830_42ac_8c65_04d21a051342.slice/crio-bcf3ce424034fa8f09a1fe9cb9f5ee361b6e9a4c463f6aefff754d43c9538b5d WatchSource:0}: Error finding container bcf3ce424034fa8f09a1fe9cb9f5ee361b6e9a4c463f6aefff754d43c9538b5d: Status 404 returned error can't find the container with id bcf3ce424034fa8f09a1fe9cb9f5ee361b6e9a4c463f6aefff754d43c9538b5d Nov 01 20:32:30.513287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:30.513247 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 20:32:30.513851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:30.513322 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 20:32:31.245355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:31.244945 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67848b78c7-ncqtf" event={"ID":"eb075952-c830-42ac-8c65-04d21a051342","Type":"ContainerStarted","Data":"9c79210e98916ff4d0667a2b7115a30802d1b927d5780b8cf1a4fc6e404d9962"} Nov 01 20:32:31.245355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:31.245007 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67848b78c7-ncqtf" event={"ID":"eb075952-c830-42ac-8c65-04d21a051342","Type":"ContainerStarted","Data":"bcf3ce424034fa8f09a1fe9cb9f5ee361b6e9a4c463f6aefff754d43c9538b5d"} Nov 01 20:32:31.375356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:31.374805 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67848b78c7-ncqtf" podStartSLOduration=2.374784005 podStartE2EDuration="2.374784005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:31.370793326 +0000 UTC m=+1759.045052582" watchObservedRunningTime="2026-11-01 20:32:31.374784005 +0000 UTC m=+1759.049043260" Nov 01 20:32:31.512380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:31.512239 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 20:32:31.512380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:31.512296 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 20:32:32.511805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:32.511767 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 20:32:32.512363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:32.511823 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 20:32:33.511854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:33.511814 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 20:32:33.512267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:33.511905 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 20:32:34.511221 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:34.511179 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 20:32:34.511479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:34.511252 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 20:32:35.582340 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:35.582305 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:35.582811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:35.582782 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:35.583710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:35.583688 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:32:39.118232 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:39.116783 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 20:32:39.838759 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:39.838717 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:39.838954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:39.838784 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:39.843132 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:39.843106 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:40.125806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:40.125715 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:32:40.127812 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:40.127785 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-54d6db5d4f-6rv48" Nov 01 20:32:40.322852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:40.322823 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67848b78c7-ncqtf" Nov 01 20:32:43.305249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.305212 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph"] Nov 01 20:32:43.309852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.309830 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.376572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376536 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.376782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376592 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.376782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376665 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.376782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376709 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.376782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376753 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.376945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376813 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.376945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376885 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.377023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.376976 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a2c36-19a0-404b-88f4-a453b3ce7513-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.377098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.377080 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4kr\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-kube-api-access-sw4kr\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.404807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.404765 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph"] Nov 01 20:32:43.478527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478489 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478537 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a2c36-19a0-404b-88f4-a453b3ce7513-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478596 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4kr\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-kube-api-access-sw4kr\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478635 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478667 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478690 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478787 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478851 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.478993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.478902 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.479155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.479008 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.479265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.479242 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.479345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.479257 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.479345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.479318 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.479423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.479360 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a2c36-19a0-404b-88f4-a453b3ce7513-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.481392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.481364 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.481502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.481424 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.535314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.535281 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.535537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.535512 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4kr\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-kube-api-access-sw4kr\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-7rhph\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:43.625424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:43.625298 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:44.063323 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:32:44.063281 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57a2c36_19a0_404b_88f4_a453b3ce7513.slice/crio-96b19f80a1ad949f493534e5eeef59357fbd473dd0d479c7e9340f57f34e54f4 WatchSource:0}: Error finding container 96b19f80a1ad949f493534e5eeef59357fbd473dd0d479c7e9340f57f34e54f4: Status 404 returned error can't find the container with id 96b19f80a1ad949f493534e5eeef59357fbd473dd0d479c7e9340f57f34e54f4 Nov 01 20:32:44.063514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.063493 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph"] Nov 01 20:32:44.066021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.065988 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:44.066122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.066059 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:44.066122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.066095 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:32:44.352951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.352860 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" event={"ID":"c57a2c36-19a0-404b-88f4-a453b3ce7513","Type":"ContainerStarted","Data":"90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8"} Nov 01 20:32:44.352951 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.352926 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" event={"ID":"c57a2c36-19a0-404b-88f4-a453b3ce7513","Type":"ContainerStarted","Data":"96b19f80a1ad949f493534e5eeef59357fbd473dd0d479c7e9340f57f34e54f4"} Nov 01 20:32:44.469414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.469362 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podStartSLOduration=1.469343579 podStartE2EDuration="1.469343579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:44.467452832 +0000 UTC m=+1772.141712098" watchObservedRunningTime="2026-11-01 20:32:44.469343579 +0000 UTC m=+1772.143602828" Nov 01 20:32:44.626192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:44.626103 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:45.627954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:45.627922 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-7rhph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 20:32:45.628454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:45.627979 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 20:32:46.158454 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:46.158415 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:32:46.625856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:46.625816 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-7rhph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 20:32:46.626050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:46.625888 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 20:32:47.626271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:47.626233 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-7rhph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 20:32:47.626911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:47.626293 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 20:32:48.625987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:48.625950 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-7rhph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 20:32:48.626173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:48.626010 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 20:32:49.628355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:49.627918 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-7rhph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 20:32:49.628355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:49.627986 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 20:32:50.630142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:50.630110 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:50.630902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:50.630603 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:50.631383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:50.631356 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:32:51.303860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.303819 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v"] Nov 01 20:32:51.308674 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.308644 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.336376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.336313 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.336579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.336455 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.336579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.336490 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmhj\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-kube-api-access-7cmhj\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.336579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.336568 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.360958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.360921 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-e385637602d5\"" Nov 01 20:32:51.361430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.361408 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\"" Nov 01 20:32:51.361929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.361908 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-4bc05b0bda98\"" Nov 01 20:32:51.362811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.362782 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-caf6c2f34a9e\"" Nov 01 20:32:51.363033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.363007 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\"" Nov 01 20:32:51.369409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.369377 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v"] Nov 01 20:32:51.407853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.407819 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\"" Nov 01 20:32:51.408131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.407874 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-t8dl8\"" Nov 01 20:32:51.408131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.407880 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-dc7c20a2c19f\"" Nov 01 20:32:51.408131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.407926 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\"" Nov 01 20:32:51.408315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.407927 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-cd18c1528ab8\"" Nov 01 20:32:51.438059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.438022 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.438249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.438074 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.438249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.438094 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmhj\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-kube-api-access-7cmhj\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.438249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.438119 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.438823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.438798 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.438929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.438840 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.441317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.441283 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.494600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.494562 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmhj\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-kube-api-access-7cmhj\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:51.624998 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:51.624896 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:52.012134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.012101 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v"] Nov 01 20:32:52.012956 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:32:52.012933 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1771e1a_c04c_4257_8cf6_deb67eefa7ac.slice/crio-9d7a94289ea28f69428fdb8b1552eb3ab6b686c40dd8366626e21c14be7cfe10 WatchSource:0}: Error finding container 9d7a94289ea28f69428fdb8b1552eb3ab6b686c40dd8366626e21c14be7cfe10: Status 404 returned error can't find the container with id 9d7a94289ea28f69428fdb8b1552eb3ab6b686c40dd8366626e21c14be7cfe10 Nov 01 20:32:52.019490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.019460 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:52.447203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.447106 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" event={"ID":"b1771e1a-c04c-4257-8cf6-deb67eefa7ac","Type":"ContainerStarted","Data":"6d9d1fee8b98feea902068e54fc4bb7492eab081cb7cd3c7227fe12a39ae1271"} Nov 01 20:32:52.447203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.447174 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" event={"ID":"b1771e1a-c04c-4257-8cf6-deb67eefa7ac","Type":"ContainerStarted","Data":"9d7a94289ea28f69428fdb8b1552eb3ab6b686c40dd8366626e21c14be7cfe10"} Nov 01 20:32:52.447468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.447223 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:32:52.563900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.563836 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" podStartSLOduration=1.5638172529999999 podStartE2EDuration="1.563817253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:52.559763092 +0000 UTC m=+1780.234022348" watchObservedRunningTime="2026-11-01 20:32:52.563817253 +0000 UTC m=+1780.238076509" Nov 01 20:32:52.876270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.876234 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc"] Nov 01 20:32:52.880786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.880768 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:52.932654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.932547 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:32:52.932654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.932622 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:32:52.932654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.932626 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:32:52.933921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.933071 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:32:52.936563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.936536 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc"] Nov 01 20:32:52.968798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.968756 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:52.969029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.968814 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:52.969029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:52.969007 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxxb\" (UniqueName: \"kubernetes.io/projected/d13cbea0-7683-4eef-9a2d-1331a07f94b8-kube-api-access-gdxxb\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.070817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.070779 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.071265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.070826 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.071265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.070928 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxxb\" (UniqueName: \"kubernetes.io/projected/d13cbea0-7683-4eef-9a2d-1331a07f94b8-kube-api-access-gdxxb\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.071554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.071533 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.073499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.073478 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.127656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.127587 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxxb\" (UniqueName: \"kubernetes.io/projected/d13cbea0-7683-4eef-9a2d-1331a07f94b8-kube-api-access-gdxxb\") pod \"maas-api-revoke-keys-e2e-mt-71335882-hskbc\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.196102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.196065 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:53.505112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.505067 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5bdbf44876-vjp4b" podUID="9e236ab5-4c60-4d6f-b98c-5a9979e0d390" containerName="console" containerID="cri-o://731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6" gracePeriod=15 Nov 01 20:32:53.579739 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.579708 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc"] Nov 01 20:32:53.588741 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:32:53.588706 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13cbea0_7683_4eef_9a2d_1331a07f94b8.slice/crio-5ece6e5f39b3e69a4617618ee32121a1afcb3202afc516e0d38e8950b3f67bf0 WatchSource:0}: Error finding container 5ece6e5f39b3e69a4617618ee32121a1afcb3202afc516e0d38e8950b3f67bf0: Status 404 returned error can't find the container with id 5ece6e5f39b3e69a4617618ee32121a1afcb3202afc516e0d38e8950b3f67bf0 Nov 01 20:32:53.818852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.818823 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:32:53.881904 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.881867 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-kube-api-access-xrfxt\" (UniqueName: \"kubernetes.io/projected/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-kube-api-access-xrfxt\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.881951 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-trusted-ca-bundle\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.881977 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-service-ca\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882025 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-tmp\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882188 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-config\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-config\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882297 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-serving-cert\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882405 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-oauth-config\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882413 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-tmp" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:53.882555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882535 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-trusted-ca-bundle" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:53.882618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882537 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-oauth-serving-cert\") pod \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\" (UID: \"9e236ab5-4c60-4d6f-b98c-5a9979e0d390\") " Nov 01 20:32:53.882669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882595 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-service-ca" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:53.882669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882616 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-config" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:53.882848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.882826 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-oauth-serving-cert" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:53.883411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.883390 5633 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-oauth-serving-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.883528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.883413 5633 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-trusted-ca-bundle\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.883528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.883427 5633 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.883528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.883438 5633 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-tmp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.883528 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.883453 5633 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-config\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.885082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.885057 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-serving-cert" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:53.885269 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.885247 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-oauth-config" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:53.885368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.885285 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-kube-api-access-xrfxt" pod "9e236ab5-4c60-4d6f-b98c-5a9979e0d390" (UID: "9e236ab5-4c60-4d6f-b98c-5a9979e0d390"). InnerVolumeSpecName "kube-api-access-xrfxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:53.984769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.984739 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrfxt\" (UniqueName: \"kubernetes.io/projected/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-kube-api-access-xrfxt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.984769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.984771 5633 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-serving-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:53.985046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:53.984786 5633 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e236ab5-4c60-4d6f-b98c-5a9979e0d390-console-oauth-config\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:54.464671 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.464632 5633 generic.go:356] "Generic (PLEG): container finished" podID="9e236ab5-4c60-4d6f-b98c-5a9979e0d390" containerID="731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6" exitCode=0 Nov 01 20:32:54.465192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.464843 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdbf44876-vjp4b" Nov 01 20:32:54.469197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.469162 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbf44876-vjp4b" event={"ID":"9e236ab5-4c60-4d6f-b98c-5a9979e0d390","Type":"ContainerDied","Data":"731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6"} Nov 01 20:32:54.469394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.469217 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdbf44876-vjp4b" event={"ID":"9e236ab5-4c60-4d6f-b98c-5a9979e0d390","Type":"ContainerDied","Data":"71e4418cdc5a889a85fcc1785006d787a38e169ac275e6742ed9924a904babc0"} Nov 01 20:32:54.469394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.469236 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" event={"ID":"d13cbea0-7683-4eef-9a2d-1331a07f94b8","Type":"ContainerStarted","Data":"06bb8d44dc46507af276eabde62c02794ce6e96e3ca77f7175bb87b69f30f29d"} Nov 01 20:32:54.469394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.469255 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" event={"ID":"d13cbea0-7683-4eef-9a2d-1331a07f94b8","Type":"ContainerStarted","Data":"5ece6e5f39b3e69a4617618ee32121a1afcb3202afc516e0d38e8950b3f67bf0"} Nov 01 20:32:54.469394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.469381 5633 scope.go:122] "RemoveContainer" containerID="731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6" Nov 01 20:32:54.484984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.484957 5633 scope.go:122] "RemoveContainer" containerID="731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6" Nov 01 20:32:54.485390 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:32:54.485323 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6\": container with ID starting with 731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6 not found: ID does not exist" containerID="731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6" Nov 01 20:32:54.485463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.485404 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6"} err="failed to get container status \"731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6\": rpc error: code = NotFound desc = could not find container \"731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6\": container with ID starting with 731410292ffce7bffa1753a558963c39592dba8f00a70be5d9408d5c065e3fb6 not found: ID does not exist" Nov 01 20:32:54.630395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.630349 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdbf44876-vjp4b"] Nov 01 20:32:54.635393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.635363 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bdbf44876-vjp4b"] Nov 01 20:32:54.745622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:54.745256 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" podStartSLOduration=2.745236723 podStartE2EDuration="2.745236723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:54.740818657 +0000 UTC m=+1782.415077920" watchObservedRunningTime="2026-11-01 20:32:54.745236723 +0000 UTC m=+1782.419495978" Nov 01 20:32:55.172449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:55.172354 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq"] Nov 01 20:32:55.200068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:55.200028 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:55.279456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:55.278550 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5"] Nov 01 20:32:55.483154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:55.483120 5633 generic.go:356] "Generic (PLEG): container finished" podID="d13cbea0-7683-4eef-9a2d-1331a07f94b8" containerID="06bb8d44dc46507af276eabde62c02794ce6e96e3ca77f7175bb87b69f30f29d" exitCode=0 Nov 01 20:32:55.483677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:55.483208 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" event={"ID":"d13cbea0-7683-4eef-9a2d-1331a07f94b8","Type":"ContainerDied","Data":"06bb8d44dc46507af276eabde62c02794ce6e96e3ca77f7175bb87b69f30f29d"} Nov 01 20:32:55.521655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:55.521621 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:55.646184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:55.646063 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:56.481248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:56.480699 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e236ab5-4c60-4d6f-b98c-5a9979e0d390" path="/var/lib/kubelet/pods/9e236ab5-4c60-4d6f-b98c-5a9979e0d390/volumes" Nov 01 20:32:57.017288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.014737 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:57.106514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.106480 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-openshift-ca\") pod \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " Nov 01 20:32:57.106701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.106531 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-xks-ca\") pod \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " Nov 01 20:32:57.106701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.106656 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13cbea0-7683-4eef-9a2d-1331a07f94b8-kube-api-access-gdxxb\" (UniqueName: \"kubernetes.io/projected/d13cbea0-7683-4eef-9a2d-1331a07f94b8-kube-api-access-gdxxb\") pod \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\" (UID: \"d13cbea0-7683-4eef-9a2d-1331a07f94b8\") " Nov 01 20:32:57.106888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.106867 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-openshift-ca" pod "d13cbea0-7683-4eef-9a2d-1331a07f94b8" (UID: "d13cbea0-7683-4eef-9a2d-1331a07f94b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:57.107491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.107470 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:57.108946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.108919 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13cbea0-7683-4eef-9a2d-1331a07f94b8-kube-api-access-gdxxb" pod "d13cbea0-7683-4eef-9a2d-1331a07f94b8" (UID: "d13cbea0-7683-4eef-9a2d-1331a07f94b8"). InnerVolumeSpecName "kube-api-access-gdxxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:57.109057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.108957 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-xks-ca" pod "d13cbea0-7683-4eef-9a2d-1331a07f94b8" (UID: "d13cbea0-7683-4eef-9a2d-1331a07f94b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:57.132047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.132010 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:32:57.209627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.209533 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxxb\" (UniqueName: \"kubernetes.io/projected/d13cbea0-7683-4eef-9a2d-1331a07f94b8-kube-api-access-gdxxb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:57.209627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.209563 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13cbea0-7683-4eef-9a2d-1331a07f94b8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:32:57.508825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.508789 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" event={"ID":"d13cbea0-7683-4eef-9a2d-1331a07f94b8","Type":"ContainerDied","Data":"5ece6e5f39b3e69a4617618ee32121a1afcb3202afc516e0d38e8950b3f67bf0"} Nov 01 20:32:57.509014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.508852 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ece6e5f39b3e69a4617618ee32121a1afcb3202afc516e0d38e8950b3f67bf0" Nov 01 20:32:57.509014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:57.508801 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc" Nov 01 20:32:59.698011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.697980 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6"] Nov 01 20:32:59.698696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.698681 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d13cbea0-7683-4eef-9a2d-1331a07f94b8" containerName="revoke-keys" Nov 01 20:32:59.698745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.698698 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13cbea0-7683-4eef-9a2d-1331a07f94b8" containerName="revoke-keys" Nov 01 20:32:59.698785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.698747 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9e236ab5-4c60-4d6f-b98c-5a9979e0d390" containerName="console" Nov 01 20:32:59.698785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.698752 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e236ab5-4c60-4d6f-b98c-5a9979e0d390" containerName="console" Nov 01 20:32:59.698870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.698861 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e236ab5-4c60-4d6f-b98c-5a9979e0d390" containerName="console" Nov 01 20:32:59.698905 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.698877 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13cbea0-7683-4eef-9a2d-1331a07f94b8" containerName="revoke-keys" Nov 01 20:32:59.702776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.702753 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.749473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.749437 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5x5\" (UniqueName: \"kubernetes.io/projected/38a4437f-50d5-4c35-98df-a88fb0a57f6d-kube-api-access-8l5x5\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.749645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.749556 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.749871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.749813 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.759077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.759047 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6"] Nov 01 20:32:59.800816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.800777 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:32:59.800816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.800805 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:32:59.803507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.803485 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:32:59.803556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.803508 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:32:59.851053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.851021 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.851269 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.851109 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5x5\" (UniqueName: \"kubernetes.io/projected/38a4437f-50d5-4c35-98df-a88fb0a57f6d-kube-api-access-8l5x5\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.851269 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.851158 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.851925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.851906 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.853802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.853766 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:32:59.907468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:32:59.907435 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5x5\" (UniqueName: \"kubernetes.io/projected/38a4437f-50d5-4c35-98df-a88fb0a57f6d-kube-api-access-8l5x5\") pod \"maas-api-revoke-keys-e2e-ait-393dc263-hzrl6\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:33:00.021838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.021805 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:33:00.172932 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.172884 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" podUID="34d9d461-cfeb-40fa-8459-e14a64b5e49d" containerName="payload-processing" containerID="cri-o://5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1" gracePeriod=25 Nov 01 20:33:00.279917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.279818 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" podUID="fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" containerName="payload-pre-processing" containerID="cri-o://9998dffd00fd6961364252b9ccafc24439560f5c343b88ae03a9a5d01be111bc" gracePeriod=25 Nov 01 20:33:00.408075 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:00.407876 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a4437f_50d5_4c35_98df_a88fb0a57f6d.slice/crio-6308e41bac88ff728939653e1cfa5602c8851f40c875350598f44dd5fdcf82df WatchSource:0}: Error finding container 6308e41bac88ff728939653e1cfa5602c8851f40c875350598f44dd5fdcf82df: Status 404 returned error can't find the container with id 6308e41bac88ff728939653e1cfa5602c8851f40c875350598f44dd5fdcf82df Nov 01 20:33:00.409114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.408903 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6"] Nov 01 20:33:00.472854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.472824 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:33:00.538077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.537977 5633 generic.go:356] "Generic (PLEG): container finished" podID="fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" containerID="9998dffd00fd6961364252b9ccafc24439560f5c343b88ae03a9a5d01be111bc" exitCode=0 Nov 01 20:33:00.538077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.538050 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" event={"ID":"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0","Type":"ContainerDied","Data":"9998dffd00fd6961364252b9ccafc24439560f5c343b88ae03a9a5d01be111bc"} Nov 01 20:33:00.540095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.540059 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" event={"ID":"38a4437f-50d5-4c35-98df-a88fb0a57f6d","Type":"ContainerStarted","Data":"6308e41bac88ff728939653e1cfa5602c8851f40c875350598f44dd5fdcf82df"} Nov 01 20:33:00.542393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.542318 5633 generic.go:356] "Generic (PLEG): container finished" podID="34d9d461-cfeb-40fa-8459-e14a64b5e49d" containerID="5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1" exitCode=0 Nov 01 20:33:00.542550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.542425 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" event={"ID":"34d9d461-cfeb-40fa-8459-e14a64b5e49d","Type":"ContainerDied","Data":"5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1"} Nov 01 20:33:00.542550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.542460 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" event={"ID":"34d9d461-cfeb-40fa-8459-e14a64b5e49d","Type":"ContainerDied","Data":"b42029fe51ede968e9f95718e23d84f4ccc1f75f725589149aadefddcf1fe9ca"} Nov 01 20:33:00.542550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.542481 5633 scope.go:122] "RemoveContainer" containerID="5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1" Nov 01 20:33:00.542724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.542630 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq" Nov 01 20:33:00.557521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.557322 5633 scope.go:122] "RemoveContainer" containerID="5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1" Nov 01 20:33:00.557807 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:00.557753 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1\": container with ID starting with 5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1 not found: ID does not exist" containerID="5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1" Nov 01 20:33:00.557807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.557789 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1"} err="failed to get container status \"5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1\": rpc error: code = NotFound desc = could not find container \"5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1\": container with ID starting with 5dc17a87e3b349d3e324278ddac4171536dfdc052c5fe56e22cb82d6b82820c1 not found: ID does not exist" Nov 01 20:33:00.563159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.563135 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d9d461-cfeb-40fa-8459-e14a64b5e49d-kube-api-access-hqql5\" (UniqueName: \"kubernetes.io/projected/34d9d461-cfeb-40fa-8459-e14a64b5e49d-kube-api-access-hqql5\") pod \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " Nov 01 20:33:00.563397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.563382 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d9d461-cfeb-40fa-8459-e14a64b5e49d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34d9d461-cfeb-40fa-8459-e14a64b5e49d-plugins-config-volume\") pod \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\" (UID: \"34d9d461-cfeb-40fa-8459-e14a64b5e49d\") " Nov 01 20:33:00.563791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.563756 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d9d461-cfeb-40fa-8459-e14a64b5e49d-plugins-config-volume" pod "34d9d461-cfeb-40fa-8459-e14a64b5e49d" (UID: "34d9d461-cfeb-40fa-8459-e14a64b5e49d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:00.564023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.563940 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34d9d461-cfeb-40fa-8459-e14a64b5e49d-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:00.586251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.586218 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d9d461-cfeb-40fa-8459-e14a64b5e49d-kube-api-access-hqql5" pod "34d9d461-cfeb-40fa-8459-e14a64b5e49d" (UID: "34d9d461-cfeb-40fa-8459-e14a64b5e49d"). InnerVolumeSpecName "kube-api-access-hqql5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:00.599437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.599408 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:33:00.664497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.664468 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-plugins-config-volume\") pod \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " Nov 01 20:33:00.664688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.664574 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-kube-api-access-stl5x\") pod \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\" (UID: \"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0\") " Nov 01 20:33:00.664928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.664906 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-plugins-config-volume" pod "fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" (UID: "fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:00.665181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.665166 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqql5\" (UniqueName: \"kubernetes.io/projected/34d9d461-cfeb-40fa-8459-e14a64b5e49d-kube-api-access-hqql5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:00.665223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.665211 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:00.666873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.666849 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-kube-api-access-stl5x" pod "fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" (UID: "fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0"). InnerVolumeSpecName "kube-api-access-stl5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:00.766862 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.766826 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0-kube-api-access-stl5x\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:00.966747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.966701 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq"] Nov 01 20:33:00.970950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:00.970921 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-6hcfq"] Nov 01 20:33:01.131914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.131835 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:01.550995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.550965 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" Nov 01 20:33:01.550995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.550985 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5" event={"ID":"fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0","Type":"ContainerDied","Data":"75ebe3fc9efc201c55c8e26d929417be0418711ed9a584514fe3d645b8f42810"} Nov 01 20:33:01.551265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.551047 5633 scope.go:122] "RemoveContainer" containerID="9998dffd00fd6961364252b9ccafc24439560f5c343b88ae03a9a5d01be111bc" Nov 01 20:33:01.553091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.553066 5633 generic.go:356] "Generic (PLEG): container finished" podID="38a4437f-50d5-4c35-98df-a88fb0a57f6d" containerID="f8b3fd590a9d69ae0abb7c2ee36c05364599880370dff4951d6bb89cff652519" exitCode=0 Nov 01 20:33:01.553201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.553127 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" event={"ID":"38a4437f-50d5-4c35-98df-a88fb0a57f6d","Type":"ContainerDied","Data":"f8b3fd590a9d69ae0abb7c2ee36c05364599880370dff4951d6bb89cff652519"} Nov 01 20:33:01.717014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.716977 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5"] Nov 01 20:33:01.764620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:01.764584 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-qbpj5"] Nov 01 20:33:02.430391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.430352 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:02.467765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.467732 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d9d461-cfeb-40fa-8459-e14a64b5e49d" path="/var/lib/kubelet/pods/34d9d461-cfeb-40fa-8459-e14a64b5e49d/volumes" Nov 01 20:33:02.468218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.468204 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" path="/var/lib/kubelet/pods/fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0/volumes" Nov 01 20:33:02.734829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.734803 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:33:02.798725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.798688 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-xks-ca\") pod \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " Nov 01 20:33:02.798906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.798844 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a4437f-50d5-4c35-98df-a88fb0a57f6d-kube-api-access-8l5x5\" (UniqueName: \"kubernetes.io/projected/38a4437f-50d5-4c35-98df-a88fb0a57f6d-kube-api-access-8l5x5\") pod \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " Nov 01 20:33:02.798906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.798873 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-openshift-ca\") pod \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\" (UID: \"38a4437f-50d5-4c35-98df-a88fb0a57f6d\") " Nov 01 20:33:02.799302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.799276 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-openshift-ca" pod "38a4437f-50d5-4c35-98df-a88fb0a57f6d" (UID: "38a4437f-50d5-4c35-98df-a88fb0a57f6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:02.799623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.799502 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:02.801071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.801044 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-xks-ca" pod "38a4437f-50d5-4c35-98df-a88fb0a57f6d" (UID: "38a4437f-50d5-4c35-98df-a88fb0a57f6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:02.801261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.801238 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a4437f-50d5-4c35-98df-a88fb0a57f6d-kube-api-access-8l5x5" pod "38a4437f-50d5-4c35-98df-a88fb0a57f6d" (UID: "38a4437f-50d5-4c35-98df-a88fb0a57f6d"). InnerVolumeSpecName "kube-api-access-8l5x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:02.901179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.901137 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a4437f-50d5-4c35-98df-a88fb0a57f6d-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:02.901179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:02.901170 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l5x5\" (UniqueName: \"kubernetes.io/projected/38a4437f-50d5-4c35-98df-a88fb0a57f6d-kube-api-access-8l5x5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:03.458564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:03.458532 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:33:03.573092 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:03.573057 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" Nov 01 20:33:03.573092 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:03.573075 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6" event={"ID":"38a4437f-50d5-4c35-98df-a88fb0a57f6d","Type":"ContainerDied","Data":"6308e41bac88ff728939653e1cfa5602c8851f40c875350598f44dd5fdcf82df"} Nov 01 20:33:03.573313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:03.573121 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6308e41bac88ff728939653e1cfa5602c8851f40c875350598f44dd5fdcf82df" Nov 01 20:33:07.386261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:07.386226 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:10.561316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:10.561279 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d"] Nov 01 20:33:10.673110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:10.673075 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz"] Nov 01 20:33:10.905309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:10.905230 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:33:10.907409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:10.907372 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:11.076139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:11.076105 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:12.133154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:12.133103 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:15.562636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:15.562588 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" podUID="636e42fc-694a-4c2a-9cfa-b3bff1c05790" containerName="payload-processing" containerID="cri-o://3acc54ab9d9c71c08dd4c0476823c959f6e7d36f36689ffd591d8b964a37b0bf" gracePeriod=25 Nov 01 20:33:15.673507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:15.673464 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" podUID="52969ce6-85f7-4a62-8204-8c42d0d0c5b6" containerName="payload-pre-processing" containerID="cri-o://b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00" gracePeriod=25 Nov 01 20:33:15.691257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:15.691223 5633 generic.go:356] "Generic (PLEG): container finished" podID="636e42fc-694a-4c2a-9cfa-b3bff1c05790" containerID="3acc54ab9d9c71c08dd4c0476823c959f6e7d36f36689ffd591d8b964a37b0bf" exitCode=0 Nov 01 20:33:15.691483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:15.691402 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" event={"ID":"636e42fc-694a-4c2a-9cfa-b3bff1c05790","Type":"ContainerDied","Data":"3acc54ab9d9c71c08dd4c0476823c959f6e7d36f36689ffd591d8b964a37b0bf"} Nov 01 20:33:15.879474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:15.879143 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:33:15.956430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:15.956394 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:33:16.055604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.055571 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636e42fc-694a-4c2a-9cfa-b3bff1c05790-kube-api-access-6xbrz\" (UniqueName: \"kubernetes.io/projected/636e42fc-694a-4c2a-9cfa-b3bff1c05790-kube-api-access-6xbrz\") pod \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " Nov 01 20:33:16.055800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.055613 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-kube-api-access-k8rcs\" (UniqueName: \"kubernetes.io/projected/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-kube-api-access-k8rcs\") pod \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " Nov 01 20:33:16.055875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.055798 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-plugins-config-volume\") pod \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\" (UID: \"52969ce6-85f7-4a62-8204-8c42d0d0c5b6\") " Nov 01 20:33:16.055937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.055891 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636e42fc-694a-4c2a-9cfa-b3bff1c05790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e42fc-694a-4c2a-9cfa-b3bff1c05790-plugins-config-volume\") pod \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\" (UID: \"636e42fc-694a-4c2a-9cfa-b3bff1c05790\") " Nov 01 20:33:16.056233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.056204 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-plugins-config-volume" pod "52969ce6-85f7-4a62-8204-8c42d0d0c5b6" (UID: "52969ce6-85f7-4a62-8204-8c42d0d0c5b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:16.056490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.056242 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636e42fc-694a-4c2a-9cfa-b3bff1c05790-plugins-config-volume" pod "636e42fc-694a-4c2a-9cfa-b3bff1c05790" (UID: "636e42fc-694a-4c2a-9cfa-b3bff1c05790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:16.056490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.056521 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636e42fc-694a-4c2a-9cfa-b3bff1c05790-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:16.056490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.056537 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:16.057997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.057970 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-kube-api-access-k8rcs" pod "52969ce6-85f7-4a62-8204-8c42d0d0c5b6" (UID: "52969ce6-85f7-4a62-8204-8c42d0d0c5b6"). InnerVolumeSpecName "kube-api-access-k8rcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:16.058093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.058001 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636e42fc-694a-4c2a-9cfa-b3bff1c05790-kube-api-access-6xbrz" pod "636e42fc-694a-4c2a-9cfa-b3bff1c05790" (UID: "636e42fc-694a-4c2a-9cfa-b3bff1c05790"). InnerVolumeSpecName "kube-api-access-6xbrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:16.158201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.158107 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xbrz\" (UniqueName: \"kubernetes.io/projected/636e42fc-694a-4c2a-9cfa-b3bff1c05790-kube-api-access-6xbrz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:16.158201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.158152 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8rcs\" (UniqueName: \"kubernetes.io/projected/52969ce6-85f7-4a62-8204-8c42d0d0c5b6-kube-api-access-k8rcs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:16.170789 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:16.170746 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:33:16.375228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375193 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-13b913"] Nov 01 20:33:16.375899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375880 5633 cpu_manager.go:414] "containerMap: removing container" podUID="34d9d461-cfeb-40fa-8459-e14a64b5e49d" containerName="payload-processing" Nov 01 20:33:16.375965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375901 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d9d461-cfeb-40fa-8459-e14a64b5e49d" containerName="payload-processing" Nov 01 20:33:16.375965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375914 5633 cpu_manager.go:414] "containerMap: removing container" podUID="38a4437f-50d5-4c35-98df-a88fb0a57f6d" containerName="revoke-keys" Nov 01 20:33:16.375965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375919 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a4437f-50d5-4c35-98df-a88fb0a57f6d" containerName="revoke-keys" Nov 01 20:33:16.375965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375938 5633 cpu_manager.go:414] "containerMap: removing container" podUID="636e42fc-694a-4c2a-9cfa-b3bff1c05790" containerName="payload-processing" Nov 01 20:33:16.375965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375943 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636e42fc-694a-4c2a-9cfa-b3bff1c05790" containerName="payload-processing" Nov 01 20:33:16.375965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375953 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" containerName="payload-pre-processing" Nov 01 20:33:16.375965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375958 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" containerName="payload-pre-processing" Nov 01 20:33:16.376200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375972 5633 cpu_manager.go:414] "containerMap: removing container" podUID="52969ce6-85f7-4a62-8204-8c42d0d0c5b6" containerName="payload-pre-processing" Nov 01 20:33:16.376200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.375976 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52969ce6-85f7-4a62-8204-8c42d0d0c5b6" containerName="payload-pre-processing" Nov 01 20:33:16.376200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.376117 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="636e42fc-694a-4c2a-9cfa-b3bff1c05790" containerName="payload-processing" Nov 01 20:33:16.376200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.376130 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe88eb6f-f71c-4f24-bb8b-51b78d57a6a0" containerName="payload-pre-processing" Nov 01 20:33:16.376200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.376143 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="52969ce6-85f7-4a62-8204-8c42d0d0c5b6" containerName="payload-pre-processing" Nov 01 20:33:16.376200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.376160 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d9d461-cfeb-40fa-8459-e14a64b5e49d" containerName="payload-processing" Nov 01 20:33:16.376200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.376168 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a4437f-50d5-4c35-98df-a88fb0a57f6d" containerName="revoke-keys" Nov 01 20:33:16.380134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.380112 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:16.431827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.431736 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wlz8j\"" Nov 01 20:33:16.433391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.433359 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-13b913"] Nov 01 20:33:16.564253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.564219 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfztm\" (UniqueName: \"kubernetes.io/projected/ace7b161-a06e-4488-a17f-3d90a759440a-kube-api-access-mfztm\") pod \"test-curl-7370-13b913\" (UID: \"ace7b161-a06e-4488-a17f-3d90a759440a\") " pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:16.654080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.654050 5633 scope.go:122] "RemoveContainer" containerID="9834fbaeb0b3bcdfc463731208ee883ca6c945a6575e1a5d0315c7da700cd128" Nov 01 20:33:16.665027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.664999 5633 scope.go:122] "RemoveContainer" containerID="ec9cc216bd1c34718e073f7641afd9f4ff3ef9b42db7244dfcca54a63e68e196" Nov 01 20:33:16.665831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.665804 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfztm\" (UniqueName: \"kubernetes.io/projected/ace7b161-a06e-4488-a17f-3d90a759440a-kube-api-access-mfztm\") pod \"test-curl-7370-13b913\" (UID: \"ace7b161-a06e-4488-a17f-3d90a759440a\") " pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:16.675831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.675807 5633 scope.go:122] "RemoveContainer" containerID="a46e2670a81d69ebdba283cb2c37eeee84ef6189a48fb163e1741062d28000d3" Nov 01 20:33:16.686883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.686859 5633 scope.go:122] "RemoveContainer" containerID="26c2c144e1af38915c60086315a7540042a3e81b6e2409544827819969d1ca7a" Nov 01 20:33:16.697220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.697196 5633 scope.go:122] "RemoveContainer" containerID="c0ab39783d0c3229f720394bdb94d8700235c3ed5b05734eee979fe6e018b4bb" Nov 01 20:33:16.704303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.704278 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" Nov 01 20:33:16.704303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.704290 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d" event={"ID":"636e42fc-694a-4c2a-9cfa-b3bff1c05790","Type":"ContainerDied","Data":"852033edf3258a3fa9abaf88a93783cc911f8123f18be81723c7f8902272a73e"} Nov 01 20:33:16.704517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.704384 5633 scope.go:122] "RemoveContainer" containerID="3acc54ab9d9c71c08dd4c0476823c959f6e7d36f36689ffd591d8b964a37b0bf" Nov 01 20:33:16.707035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.707012 5633 generic.go:356] "Generic (PLEG): container finished" podID="52969ce6-85f7-4a62-8204-8c42d0d0c5b6" containerID="b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00" exitCode=0 Nov 01 20:33:16.707150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.707065 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" Nov 01 20:33:16.707150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.707070 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" event={"ID":"52969ce6-85f7-4a62-8204-8c42d0d0c5b6","Type":"ContainerDied","Data":"b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00"} Nov 01 20:33:16.707150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.707114 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz" event={"ID":"52969ce6-85f7-4a62-8204-8c42d0d0c5b6","Type":"ContainerDied","Data":"3c4db9dd73c9e0f2bb9ac114ebbd43d781685771571130cc7f57021cb813a4af"} Nov 01 20:33:16.710150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.710129 5633 scope.go:122] "RemoveContainer" containerID="255f1303c3237d22472abf07b7a3386e3db51a3b1f2820b9cc2bdd296254df29" Nov 01 20:33:16.721435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.721399 5633 scope.go:122] "RemoveContainer" containerID="b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00" Nov 01 20:33:16.722897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.722809 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfztm\" (UniqueName: \"kubernetes.io/projected/ace7b161-a06e-4488-a17f-3d90a759440a-kube-api-access-mfztm\") pod \"test-curl-7370-13b913\" (UID: \"ace7b161-a06e-4488-a17f-3d90a759440a\") " pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:16.748301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.748164 5633 scope.go:122] "RemoveContainer" containerID="b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00" Nov 01 20:33:16.748520 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:16.748479 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00\": container with ID starting with b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00 not found: ID does not exist" containerID="b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00" Nov 01 20:33:16.748572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.748517 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00"} err="failed to get container status \"b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00\": rpc error: code = NotFound desc = could not find container \"b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00\": container with ID starting with b31daa769a521c04cac26ed033eb168873e0b540e816dd4041917f3899d1ac00 not found: ID does not exist" Nov 01 20:33:16.844032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.843997 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:16.868756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.868718 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d"] Nov 01 20:33:16.873523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.873493 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-393dc263-7cc8b5f84f-dkq8d"] Nov 01 20:33:16.982061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.982022 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz"] Nov 01 20:33:16.986448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.986420 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-393dc263-5cffc8ff-bbcwz"] Nov 01 20:33:16.995166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.995143 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:16.998732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:16.998701 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:17.156785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:17.151967 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:17.374586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:17.374552 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-13b913"] Nov 01 20:33:17.375098 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:17.375061 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podace7b161_a06e_4488_a17f_3d90a759440a.slice/crio-8c459e3b651573f9149c609ee1d704d8255139e78b74413798e8c264fc700eff WatchSource:0}: Error finding container 8c459e3b651573f9149c609ee1d704d8255139e78b74413798e8c264fc700eff: Status 404 returned error can't find the container with id 8c459e3b651573f9149c609ee1d704d8255139e78b74413798e8c264fc700eff Nov 01 20:33:17.716271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:17.716238 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-13b913" event={"ID":"ace7b161-a06e-4488-a17f-3d90a759440a","Type":"ContainerStarted","Data":"04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b"} Nov 01 20:33:17.716752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:17.716287 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-13b913" event={"ID":"ace7b161-a06e-4488-a17f-3d90a759440a","Type":"ContainerStarted","Data":"8c459e3b651573f9149c609ee1d704d8255139e78b74413798e8c264fc700eff"} Nov 01 20:33:17.831293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:17.831236 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-13b913" podStartSLOduration=1.831219548 podStartE2EDuration="1.831219548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:17.827590727 +0000 UTC m=+1805.501849987" watchObservedRunningTime="2026-11-01 20:33:17.831219548 +0000 UTC m=+1805.505478803" Nov 01 20:33:18.468090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:18.468051 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52969ce6-85f7-4a62-8204-8c42d0d0c5b6" path="/var/lib/kubelet/pods/52969ce6-85f7-4a62-8204-8c42d0d0c5b6/volumes" Nov 01 20:33:18.468562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:18.468546 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636e42fc-694a-4c2a-9cfa-b3bff1c05790" path="/var/lib/kubelet/pods/636e42fc-694a-4c2a-9cfa-b3bff1c05790/volumes" Nov 01 20:33:19.178192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.178161 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:19.367586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.366837 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:19.378625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.375375 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-13b913"] Nov 01 20:33:19.387265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.387212 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-13b913"] Nov 01 20:33:19.467220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.467188 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-0a735e"] Nov 01 20:33:19.467866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.467850 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ace7b161-a06e-4488-a17f-3d90a759440a" containerName="test-curl-7370-13b913" Nov 01 20:33:19.467912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.467868 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace7b161-a06e-4488-a17f-3d90a759440a" containerName="test-curl-7370-13b913" Nov 01 20:33:19.467997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.467988 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace7b161-a06e-4488-a17f-3d90a759440a" containerName="test-curl-7370-13b913" Nov 01 20:33:19.472813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.472759 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:19.515155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.514865 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv464\" (UniqueName: \"kubernetes.io/projected/ff25f7b4-556f-413d-95b8-97dfc18c002a-kube-api-access-tv464\") pod \"test-curl-7370-0a735e\" (UID: \"ff25f7b4-556f-413d-95b8-97dfc18c002a\") " pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:19.596678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.596648 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-0a735e"] Nov 01 20:33:19.602265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.602232 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl"] Nov 01 20:33:19.606534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.606510 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.619293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.619264 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv464\" (UniqueName: \"kubernetes.io/projected/ff25f7b4-556f-413d-95b8-97dfc18c002a-kube-api-access-tv464\") pod \"test-curl-7370-0a735e\" (UID: \"ff25f7b4-556f-413d-95b8-97dfc18c002a\") " pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:19.619464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.619369 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28207c7-f498-43ed-b16f-5eeea8fd28d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.619592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.619577 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45kwj\" (UniqueName: \"kubernetes.io/projected/b28207c7-f498-43ed-b16f-5eeea8fd28d5-kube-api-access-45kwj\") pod \"payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.658438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.658402 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f154f7\"" Nov 01 20:33:19.661734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.661706 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl"] Nov 01 20:33:19.718411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.717917 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f154f7-dockercfg-bg7cm\"" Nov 01 20:33:19.750355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.745656 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45kwj\" (UniqueName: \"kubernetes.io/projected/b28207c7-f498-43ed-b16f-5eeea8fd28d5-kube-api-access-45kwj\") pod \"payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.750355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.745896 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28207c7-f498-43ed-b16f-5eeea8fd28d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.750355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.747074 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28207c7-f498-43ed-b16f-5eeea8fd28d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.786357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.785902 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv464\" (UniqueName: \"kubernetes.io/projected/ff25f7b4-556f-413d-95b8-97dfc18c002a-kube-api-access-tv464\") pod \"test-curl-7370-0a735e\" (UID: \"ff25f7b4-556f-413d-95b8-97dfc18c002a\") " pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:19.841409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.839416 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45kwj\" (UniqueName: \"kubernetes.io/projected/b28207c7-f498-43ed-b16f-5eeea8fd28d5-kube-api-access-45kwj\") pod \"payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.851158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.849495 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t"] Nov 01 20:33:19.853022 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.852990 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:19.888958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.888886 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-13b913" podUID="ace7b161-a06e-4488-a17f-3d90a759440a" containerName="test-curl-7370-13b913" containerID="cri-o://04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b" gracePeriod=2 Nov 01 20:33:19.891253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.891226 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:19.896251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.896226 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t"] Nov 01 20:33:19.921668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.921633 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:19.941648 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:19.941607 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-13b913\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="ace7b161-a06e-4488-a17f-3d90a759440a" pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:19.986523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.986413 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ccbde2-d089-47b0-8582-e8df179b9b5f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:19.986711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:19.986639 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bpmp\" (UniqueName: \"kubernetes.io/projected/73ccbde2-d089-47b0-8582-e8df179b9b5f-kube-api-access-2bpmp\") pod \"payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:20.089008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.088965 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bpmp\" (UniqueName: \"kubernetes.io/projected/73ccbde2-d089-47b0-8582-e8df179b9b5f-kube-api-access-2bpmp\") pod \"payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:20.090404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.090373 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ccbde2-d089-47b0-8582-e8df179b9b5f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:20.091463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.091364 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ccbde2-d089-47b0-8582-e8df179b9b5f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:20.104318 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:20.104242 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-13b913\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="ace7b161-a06e-4488-a17f-3d90a759440a" pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:20.146539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.146505 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bpmp\" (UniqueName: \"kubernetes.io/projected/73ccbde2-d089-47b0-8582-e8df179b9b5f-kube-api-access-2bpmp\") pod \"payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:20.234712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.234671 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:20.266610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.266576 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-0a735e"] Nov 01 20:33:20.267013 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:20.266987 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff25f7b4_556f_413d_95b8_97dfc18c002a.slice/crio-f90d8e825c6d31d6987793637e1a4ab5b5b19f6d7cfa02aa903eb2eb9fa56150 WatchSource:0}: Error finding container f90d8e825c6d31d6987793637e1a4ab5b5b19f6d7cfa02aa903eb2eb9fa56150: Status 404 returned error can't find the container with id f90d8e825c6d31d6987793637e1a4ab5b5b19f6d7cfa02aa903eb2eb9fa56150 Nov 01 20:33:20.329986 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:20.328980 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28207c7_f498_43ed_b16f_5eeea8fd28d5.slice/crio-5eefebddaccc4d56ca4b6f4480dc6c95b5f999a7438798ca9a947d49c57241c9 WatchSource:0}: Error finding container 5eefebddaccc4d56ca4b6f4480dc6c95b5f999a7438798ca9a947d49c57241c9: Status 404 returned error can't find the container with id 5eefebddaccc4d56ca4b6f4480dc6c95b5f999a7438798ca9a947d49c57241c9 Nov 01 20:33:20.329986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.329745 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl"] Nov 01 20:33:20.617179 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:20.617150 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ccbde2_d089_47b0_8582_e8df179b9b5f.slice/crio-67368e1a4f07ea72b72d75213c2ec4a29894f2c1ae0448a12850958e8eb075a7 WatchSource:0}: Error finding container 67368e1a4f07ea72b72d75213c2ec4a29894f2c1ae0448a12850958e8eb075a7: Status 404 returned error can't find the container with id 67368e1a4f07ea72b72d75213c2ec4a29894f2c1ae0448a12850958e8eb075a7 Nov 01 20:33:20.617661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.617640 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t"] Nov 01 20:33:20.899048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.898951 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-0a735e" event={"ID":"ff25f7b4-556f-413d-95b8-97dfc18c002a","Type":"ContainerStarted","Data":"4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78"} Nov 01 20:33:20.899048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.898995 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-0a735e" event={"ID":"ff25f7b4-556f-413d-95b8-97dfc18c002a","Type":"ContainerStarted","Data":"f90d8e825c6d31d6987793637e1a4ab5b5b19f6d7cfa02aa903eb2eb9fa56150"} Nov 01 20:33:20.900732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.900705 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" event={"ID":"b28207c7-f498-43ed-b16f-5eeea8fd28d5","Type":"ContainerStarted","Data":"0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6"} Nov 01 20:33:20.900875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.900750 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" event={"ID":"b28207c7-f498-43ed-b16f-5eeea8fd28d5","Type":"ContainerStarted","Data":"5eefebddaccc4d56ca4b6f4480dc6c95b5f999a7438798ca9a947d49c57241c9"} Nov 01 20:33:20.900875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.900777 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:20.902518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.902492 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" event={"ID":"73ccbde2-d089-47b0-8582-e8df179b9b5f","Type":"ContainerStarted","Data":"a298d807f326f2facf2f7d0ad945f33ebb63edae3a85ddb6392404cb28e310dd"} Nov 01 20:33:20.902627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.902538 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:20.902627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:20.902555 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" event={"ID":"73ccbde2-d089-47b0-8582-e8df179b9b5f","Type":"ContainerStarted","Data":"67368e1a4f07ea72b72d75213c2ec4a29894f2c1ae0448a12850958e8eb075a7"} Nov 01 20:33:21.013481 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.013388 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-0a735e" podStartSLOduration=2.013373181 podStartE2EDuration="2.013373181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:21.008836853 +0000 UTC m=+1808.683096109" watchObservedRunningTime="2026-11-01 20:33:21.013373181 +0000 UTC m=+1808.687632435" Nov 01 20:33:21.122499 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.122444 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" podStartSLOduration=2.122429741 podStartE2EDuration="2.122429741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:21.121348953 +0000 UTC m=+1808.795608206" watchObservedRunningTime="2026-11-01 20:33:21.122429741 +0000 UTC m=+1808.796688996" Nov 01 20:33:21.236376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.236242 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" podStartSLOduration=2.236218856 podStartE2EDuration="2.236218856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:21.23567436 +0000 UTC m=+1808.909933616" watchObservedRunningTime="2026-11-01 20:33:21.236218856 +0000 UTC m=+1808.910478113" Nov 01 20:33:21.616683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.616587 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp"] Nov 01 20:33:21.616948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.616866 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" containerID="cri-o://7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31" gracePeriod=30 Nov 01 20:33:21.621921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.621891 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c"] Nov 01 20:33:21.622123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.622101 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" containerID="cri-o://e9946db3dfc77055b8a3f9b187508b432ff74b13450324aa341e5b75b0a7f9f7" gracePeriod=30 Nov 01 20:33:21.626671 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.626641 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7"] Nov 01 20:33:21.626918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.626887 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" containerID="cri-o://a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892" gracePeriod=30 Nov 01 20:33:21.631153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.631125 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w"] Nov 01 20:33:21.631372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.631350 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" containerID="cri-o://4a0a88b9682093d135b969685a90c29360f3d4737ee9cf28feec15cc2c8f2c5e" gracePeriod=30 Nov 01 20:33:21.833911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:21.833877 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:22.186577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.186544 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:22.331622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.331575 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace7b161-a06e-4488-a17f-3d90a759440a-kube-api-access-mfztm\" (UniqueName: \"kubernetes.io/projected/ace7b161-a06e-4488-a17f-3d90a759440a-kube-api-access-mfztm\") pod \"ace7b161-a06e-4488-a17f-3d90a759440a\" (UID: \"ace7b161-a06e-4488-a17f-3d90a759440a\") " Nov 01 20:33:22.333981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.333950 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace7b161-a06e-4488-a17f-3d90a759440a-kube-api-access-mfztm" pod "ace7b161-a06e-4488-a17f-3d90a759440a" (UID: "ace7b161-a06e-4488-a17f-3d90a759440a"). InnerVolumeSpecName "kube-api-access-mfztm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:22.426168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.426068 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-0a735e"] Nov 01 20:33:22.432596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.432557 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-0a735e"] Nov 01 20:33:22.433428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.433398 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfztm\" (UniqueName: \"kubernetes.io/projected/ace7b161-a06e-4488-a17f-3d90a759440a-kube-api-access-mfztm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:22.469031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.468999 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace7b161-a06e-4488-a17f-3d90a759440a" path="/var/lib/kubelet/pods/ace7b161-a06e-4488-a17f-3d90a759440a/volumes" Nov 01 20:33:22.921014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.920976 5633 generic.go:356] "Generic (PLEG): container finished" podID="ace7b161-a06e-4488-a17f-3d90a759440a" containerID="04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b" exitCode=137 Nov 01 20:33:22.921227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.921022 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-13b913" Nov 01 20:33:22.921227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.921076 5633 scope.go:122] "RemoveContainer" containerID="04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b" Nov 01 20:33:22.921381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.921315 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-0a735e" podUID="ff25f7b4-556f-413d-95b8-97dfc18c002a" containerName="test-curl-7370-0a735e" containerID="cri-o://4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78" gracePeriod=2 Nov 01 20:33:22.931464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.931435 5633 scope.go:122] "RemoveContainer" containerID="04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b" Nov 01 20:33:22.931787 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:22.931761 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b\": container with ID starting with 04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b not found: ID does not exist" containerID="04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b" Nov 01 20:33:22.931882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:22.931795 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b"} err="failed to get container status \"04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b\": rpc error: code = NotFound desc = could not find container \"04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b\": container with ID starting with 04e535adaada28ef5374640d7b181e7ae0a4c1e0edcc846de5aa8aad271d880b not found: ID does not exist" Nov 01 20:33:23.309074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.309035 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:23.309230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.309099 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:23.322484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.322446 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:23.322639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.322515 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:23.450650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.450616 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2"] Nov 01 20:33:23.451265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.451252 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ff25f7b4-556f-413d-95b8-97dfc18c002a" containerName="test-curl-7370-0a735e" Nov 01 20:33:23.451352 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.451267 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff25f7b4-556f-413d-95b8-97dfc18c002a" containerName="test-curl-7370-0a735e" Nov 01 20:33:23.451425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.451414 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff25f7b4-556f-413d-95b8-97dfc18c002a" containerName="test-curl-7370-0a735e" Nov 01 20:33:23.456620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.456598 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.511191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.511150 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2"] Nov 01 20:33:23.550068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.550030 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.550231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.550163 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.550231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.550225 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tr5\" (UniqueName: \"kubernetes.io/projected/3a53af7e-5492-4d2a-96d2-830f9387e5a4-kube-api-access-t2tr5\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.554095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.554068 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:33:23.554095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.554092 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:33:23.554273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.554076 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:33:23.554273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.554107 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:33:23.651807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.651727 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.651807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.651782 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.651995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.651809 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tr5\" (UniqueName: \"kubernetes.io/projected/3a53af7e-5492-4d2a-96d2-830f9387e5a4-kube-api-access-t2tr5\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.652417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.652381 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.654530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.654499 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.718459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.718428 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tr5\" (UniqueName: \"kubernetes.io/projected/3a53af7e-5492-4d2a-96d2-830f9387e5a4-kube-api-access-t2tr5\") pod \"maas-api-revoke-sub-5bb3dece-t7mw2\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:23.771948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:23.771916 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:24.147757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:24.147724 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2"] Nov 01 20:33:24.147902 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:24.147786 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a53af7e_5492_4d2a_96d2_830f9387e5a4.slice/crio-98aecabce43625b3e897d81081decfa6d70b2c0bc3b77519a8174ba95cac7dd9 WatchSource:0}: Error finding container 98aecabce43625b3e897d81081decfa6d70b2c0bc3b77519a8174ba95cac7dd9: Status 404 returned error can't find the container with id 98aecabce43625b3e897d81081decfa6d70b2c0bc3b77519a8174ba95cac7dd9 Nov 01 20:33:24.316693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:24.316655 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:24.316859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:24.316717 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:24.418138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:24.418094 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:24.940392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:24.940259 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" event={"ID":"3a53af7e-5492-4d2a-96d2-830f9387e5a4","Type":"ContainerStarted","Data":"bdd237a1a981cbb385b22231836795cc94a3eb52e3146cdf8b2f40e7faee1f1e"} Nov 01 20:33:24.940392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:24.940323 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" event={"ID":"3a53af7e-5492-4d2a-96d2-830f9387e5a4","Type":"ContainerStarted","Data":"98aecabce43625b3e897d81081decfa6d70b2c0bc3b77519a8174ba95cac7dd9"} Nov 01 20:33:25.169775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.169745 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:25.282682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.282652 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff25f7b4-556f-413d-95b8-97dfc18c002a-kube-api-access-tv464\" (UniqueName: \"kubernetes.io/projected/ff25f7b4-556f-413d-95b8-97dfc18c002a-kube-api-access-tv464\") pod \"ff25f7b4-556f-413d-95b8-97dfc18c002a\" (UID: \"ff25f7b4-556f-413d-95b8-97dfc18c002a\") " Nov 01 20:33:25.284879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.284857 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff25f7b4-556f-413d-95b8-97dfc18c002a-kube-api-access-tv464" pod "ff25f7b4-556f-413d-95b8-97dfc18c002a" (UID: "ff25f7b4-556f-413d-95b8-97dfc18c002a"). InnerVolumeSpecName "kube-api-access-tv464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:25.384450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.384409 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv464\" (UniqueName: \"kubernetes.io/projected/ff25f7b4-556f-413d-95b8-97dfc18c002a-kube-api-access-tv464\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:25.961617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.961591 5633 generic.go:356] "Generic (PLEG): container finished" podID="ff25f7b4-556f-413d-95b8-97dfc18c002a" containerID="4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78" exitCode=137 Nov 01 20:33:25.962083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.961674 5633 scope.go:122] "RemoveContainer" containerID="4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78" Nov 01 20:33:25.962083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.961735 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:25.964470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.964441 5633 generic.go:356] "Generic (PLEG): container finished" podID="3a53af7e-5492-4d2a-96d2-830f9387e5a4" containerID="bdd237a1a981cbb385b22231836795cc94a3eb52e3146cdf8b2f40e7faee1f1e" exitCode=0 Nov 01 20:33:25.964613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.964566 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" event={"ID":"3a53af7e-5492-4d2a-96d2-830f9387e5a4","Type":"ContainerDied","Data":"bdd237a1a981cbb385b22231836795cc94a3eb52e3146cdf8b2f40e7faee1f1e"} Nov 01 20:33:25.974102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.974075 5633 scope.go:122] "RemoveContainer" containerID="4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78" Nov 01 20:33:25.974865 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:25.974830 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78\": container with ID starting with 4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78 not found: ID does not exist" containerID="4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78" Nov 01 20:33:25.974985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:25.974873 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78"} err="failed to get container status \"4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78\": rpc error: code = NotFound desc = could not find container \"4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78\": container with ID starting with 4e95d81a1983fe4a57ee52a6c59bdead69d808624ff222591e5b603d779a8e78 not found: ID does not exist" Nov 01 20:33:26.126794 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:26.126736 5633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-0a735e\" is forbidden: User \"system:node:ip-10-0-128-78.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-78.us-west-1.compute.internal' and this object" podUID="ff25f7b4-556f-413d-95b8-97dfc18c002a" pod="opendatahub/test-curl-7370-0a735e" Nov 01 20:33:26.468600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.468565 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff25f7b4-556f-413d-95b8-97dfc18c002a" path="/var/lib/kubelet/pods/ff25f7b4-556f-413d-95b8-97dfc18c002a/volumes" Nov 01 20:33:26.499177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.499127 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:26.958590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.958559 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:33:26.980502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.980419 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:33:26.982349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.982293 5633 generic.go:356] "Generic (PLEG): container finished" podID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerID="7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31" exitCode=0 Nov 01 20:33:26.982504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.982398 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" event={"ID":"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed","Type":"ContainerDied","Data":"7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31"} Nov 01 20:33:26.982504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.982424 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" Nov 01 20:33:26.982504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.982439 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp" event={"ID":"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed","Type":"ContainerDied","Data":"6ac442d5f10be8f19b54f1386c3497c749071839a30b41f222a33651cac7ced6"} Nov 01 20:33:26.982504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.982460 5633 scope.go:122] "RemoveContainer" containerID="7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31" Nov 01 20:33:26.984752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.984723 5633 generic.go:356] "Generic (PLEG): container finished" podID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerID="a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892" exitCode=0 Nov 01 20:33:26.985032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.984934 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" event={"ID":"f21d6128-3b0a-4d41-828c-ed92bf4611a5","Type":"ContainerDied","Data":"a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892"} Nov 01 20:33:26.985129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.985006 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" Nov 01 20:33:26.985129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.985078 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7" event={"ID":"f21d6128-3b0a-4d41-828c-ed92bf4611a5","Type":"ContainerDied","Data":"42c3cf826021f1ec623b69b983f94a6acea86a68737152b773665f79925d28b2"} Nov 01 20:33:26.988024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.987975 5633 generic.go:356] "Generic (PLEG): container finished" podID="5696e679-5d84-4352-8122-42262f1457e0" containerID="4a0a88b9682093d135b969685a90c29360f3d4737ee9cf28feec15cc2c8f2c5e" exitCode=0 Nov 01 20:33:26.988165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.988051 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" event={"ID":"5696e679-5d84-4352-8122-42262f1457e0","Type":"ContainerDied","Data":"4a0a88b9682093d135b969685a90c29360f3d4737ee9cf28feec15cc2c8f2c5e"} Nov 01 20:33:26.990508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.990481 5633 generic.go:356] "Generic (PLEG): container finished" podID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerID="e9946db3dfc77055b8a3f9b187508b432ff74b13450324aa341e5b75b0a7f9f7" exitCode=0 Nov 01 20:33:26.990715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:26.990689 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" event={"ID":"a2817334-701e-455f-9874-0ed3d1a2cc13","Type":"ContainerDied","Data":"e9946db3dfc77055b8a3f9b187508b432ff74b13450324aa341e5b75b0a7f9f7"} Nov 01 20:33:27.004522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.004493 5633 scope.go:122] "RemoveContainer" containerID="7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31" Nov 01 20:33:27.005401 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:27.004975 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31\": container with ID starting with 7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31 not found: ID does not exist" containerID="7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31" Nov 01 20:33:27.005401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.005019 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31"} err="failed to get container status \"7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31\": rpc error: code = NotFound desc = could not find container \"7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31\": container with ID starting with 7df30e65793776107bc98c33ce5294a4f62fe9574dd23ef36f59bf7cbb3aae31 not found: ID does not exist" Nov 01 20:33:27.005401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.005046 5633 scope.go:122] "RemoveContainer" containerID="a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892" Nov 01 20:33:27.027228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.026996 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-certs\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.027621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027568 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-certs" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.027847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027613 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-podinfo\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.027847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027670 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istiod-ca-cert\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.027847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027769 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-certs\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.027847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027810 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-token\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-token\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.028059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027885 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-envoy\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.028059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027920 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-token\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-token\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.028059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.027961 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-data\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.028059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.028023 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-kube-api-access-4z8d8\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-kube-api-access-4z8d8\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.028267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.028070 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-credential-socket\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.028267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.028102 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-kube-api-access-8n5rs\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-kube-api-access-8n5rs\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.028267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.028148 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-socket\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.029078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.028817 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.029171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.029092 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-socket" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.031250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.031039 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-certs" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.032033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.031414 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-credential-socket" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.032318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.032280 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istiod-ca-cert" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:27.032493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.032462 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-data" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.033024 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.032999 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-kube-api-access-4z8d8" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "kube-api-access-4z8d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.033188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.033165 5633 scope.go:122] "RemoveContainer" containerID="a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892" Nov 01 20:33:27.033398 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.033366 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-podinfo" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:27.033672 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:27.033633 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892\": container with ID starting with a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892 not found: ID does not exist" containerID="a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892" Nov 01 20:33:27.033766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.033672 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892"} err="failed to get container status \"a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892\": rpc error: code = NotFound desc = could not find container \"a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892\": container with ID starting with a530534b89a0061ed765c95a9910b4b8ebbe70fc515adfd58f522eabb97f7892 not found: ID does not exist" Nov 01 20:33:27.033766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.033698 5633 scope.go:122] "RemoveContainer" containerID="215a2af3ae678256d293364fb9e6496c5a3fb45ef7f8e943c92bfb160b2fc0e5" Nov 01 20:33:27.035002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.034879 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-kube-api-access-8n5rs" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "kube-api-access-8n5rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.035155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.035008 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-token" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.035229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.035202 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-envoy" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.036573 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.036549 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-token" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.056854 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.056824 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:33:27.061731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.061701 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.131754 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-credential-socket\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.131839 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-data\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.131868 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-envoy\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.131904 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-kube-api-access-p5wsl\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.131955 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-istio-token\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-istio-token\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.131980 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-socket\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132067 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-envoy\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132136 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-credential-socket\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132189 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-podinfo\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132234 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-certs\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132267 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-token\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-token\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132294 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-data\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132352 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-data\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132383 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-podinfo\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132425 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-socket\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132461 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istiod-ca-cert\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132495 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5696e679-5d84-4352-8122-42262f1457e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5696e679-5d84-4352-8122-42262f1457e0-istiod-ca-cert\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132551 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-envoy\") pod \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\" (UID: \"f21d6128-3b0a-4d41-828c-ed92bf4611a5\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132581 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-credential-socket\") pod \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\" (UID: \"f087cc7b-8d2d-4d9c-a057-36ad1f1863ed\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132612 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-socket\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132703 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2817334-701e-455f-9874-0ed3d1a2cc13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2817334-701e-455f-9874-0ed3d1a2cc13-istiod-ca-cert\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132748 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-certs\") pod \"a2817334-701e-455f-9874-0ed3d1a2cc13\" (UID: \"a2817334-701e-455f-9874-0ed3d1a2cc13\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132803 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-data" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132822 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5696e679-5d84-4352-8122-42262f1457e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5696e679-5d84-4352-8122-42262f1457e0-istio-podinfo\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.132871 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-kube-api-access-dk249\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-kube-api-access-dk249\") pod \"5696e679-5d84-4352-8122-42262f1457e0\" (UID: \"5696e679-5d84-4352-8122-42262f1457e0\") " Nov 01 20:33:27.133654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133407 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-certs" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133783 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133806 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n5rs\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-kube-api-access-8n5rs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133823 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133835 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133848 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133860 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133873 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133887 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133901 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133914 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133928 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133941 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.133955 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8d8\" (UniqueName: \"kubernetes.io/projected/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-kube-api-access-4z8d8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.134264 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-credential-socket" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.134604 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-credential-socket" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.134870 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-data" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.135736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.135081 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-socket" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.136736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.135970 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-credential-socket" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.136736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.136273 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-data" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.136736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.136620 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2817334-701e-455f-9874-0ed3d1a2cc13-istiod-ca-cert" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:27.136906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.136846 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-socket" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.137312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.136955 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istiod-ca-cert" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:27.137312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.137069 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-certs" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.138390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.138363 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-socket" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.140667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.140616 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5696e679-5d84-4352-8122-42262f1457e0-istiod-ca-cert" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:27.142315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.142125 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-kube-api-access-dk249" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "kube-api-access-dk249". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.142315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.142207 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-istio-token" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.142315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.142291 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-envoy" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.142725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.142681 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-token" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.142821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.142762 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-envoy" pod "f21d6128-3b0a-4d41-828c-ed92bf4611a5" (UID: "f21d6128-3b0a-4d41-828c-ed92bf4611a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.142991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.142961 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-podinfo" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:27.142991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.142968 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-podinfo" pod "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" (UID: "f087cc7b-8d2d-4d9c-a057-36ad1f1863ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:27.143139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.143104 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-kube-api-access-p5wsl" pod "a2817334-701e-455f-9874-0ed3d1a2cc13" (UID: "a2817334-701e-455f-9874-0ed3d1a2cc13"). InnerVolumeSpecName "kube-api-access-p5wsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.144240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.144213 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-envoy" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:27.144775 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.144741 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5696e679-5d84-4352-8122-42262f1457e0-istio-podinfo" pod "5696e679-5d84-4352-8122-42262f1457e0" (UID: "5696e679-5d84-4352-8122-42262f1457e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:27.164936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.164905 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:27.235231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235147 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-xks-ca\") pod \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " Nov 01 20:33:27.235231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235206 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-openshift-ca\") pod \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " Nov 01 20:33:27.235531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235464 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a53af7e-5492-4d2a-96d2-830f9387e5a4-kube-api-access-t2tr5\" (UniqueName: \"kubernetes.io/projected/3a53af7e-5492-4d2a-96d2-830f9387e5a4-kube-api-access-t2tr5\") pod \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\" (UID: \"3a53af7e-5492-4d2a-96d2-830f9387e5a4\") " Nov 01 20:33:27.235627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235603 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-openshift-ca" pod "3a53af7e-5492-4d2a-96d2-830f9387e5a4" (UID: "3a53af7e-5492-4d2a-96d2-830f9387e5a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:27.235955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235933 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235958 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235969 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235978 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235986 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.235995 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236003 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5696e679-5d84-4352-8122-42262f1457e0-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236011 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21d6128-3b0a-4d41-828c-ed92bf4611a5-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236021 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236035 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236044 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2817334-701e-455f-9874-0ed3d1a2cc13-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236053 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236061 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5696e679-5d84-4352-8122-42262f1457e0-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236069 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk249\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-kube-api-access-dk249\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236079 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236087 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236096 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236107 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wsl\" (UniqueName: \"kubernetes.io/projected/a2817334-701e-455f-9874-0ed3d1a2cc13-kube-api-access-p5wsl\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236119 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5696e679-5d84-4352-8122-42262f1457e0-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236127 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236136 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236145 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5696e679-5d84-4352-8122-42262f1457e0-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.236212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.236153 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2817334-701e-455f-9874-0ed3d1a2cc13-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.237455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.237438 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-xks-ca" pod "3a53af7e-5492-4d2a-96d2-830f9387e5a4" (UID: "3a53af7e-5492-4d2a-96d2-830f9387e5a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:27.237895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.237875 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a53af7e-5492-4d2a-96d2-830f9387e5a4-kube-api-access-t2tr5" pod "3a53af7e-5492-4d2a-96d2-830f9387e5a4" (UID: "3a53af7e-5492-4d2a-96d2-830f9387e5a4"). InnerVolumeSpecName "kube-api-access-t2tr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:27.336959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.336925 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a53af7e-5492-4d2a-96d2-830f9387e5a4-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.336959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.336955 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tr5\" (UniqueName: \"kubernetes.io/projected/3a53af7e-5492-4d2a-96d2-830f9387e5a4-kube-api-access-t2tr5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:27.753757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.753715 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp"] Nov 01 20:33:27.760268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.760233 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-7pbbp"] Nov 01 20:33:27.778808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.778777 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:27.867825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.867790 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7"] Nov 01 20:33:27.874650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.874622 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-rxbm7"] Nov 01 20:33:27.999839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.999807 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" event={"ID":"5696e679-5d84-4352-8122-42262f1457e0","Type":"ContainerDied","Data":"fbfe5e26056bd645970fde2e83bb6a8590814bb8762a7523ad25b054391bfdc9"} Nov 01 20:33:28.000356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.999813 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w" Nov 01 20:33:28.000356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:27.999871 5633 scope.go:122] "RemoveContainer" containerID="4a0a88b9682093d135b969685a90c29360f3d4737ee9cf28feec15cc2c8f2c5e" Nov 01 20:33:28.001554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.001538 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" Nov 01 20:33:28.001554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.001544 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c" event={"ID":"a2817334-701e-455f-9874-0ed3d1a2cc13","Type":"ContainerDied","Data":"0561e514f20ee8b7c5c5065e3057134a9f55313f163708a30b9be1cfa48c4938"} Nov 01 20:33:28.003320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.003291 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" event={"ID":"3a53af7e-5492-4d2a-96d2-830f9387e5a4","Type":"ContainerDied","Data":"98aecabce43625b3e897d81081decfa6d70b2c0bc3b77519a8174ba95cac7dd9"} Nov 01 20:33:28.003483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.003374 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98aecabce43625b3e897d81081decfa6d70b2c0bc3b77519a8174ba95cac7dd9" Nov 01 20:33:28.003483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.003320 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2" Nov 01 20:33:28.014919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.014898 5633 scope.go:122] "RemoveContainer" containerID="e9946db3dfc77055b8a3f9b187508b432ff74b13450324aa341e5b75b0a7f9f7" Nov 01 20:33:28.165192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.165159 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w"] Nov 01 20:33:28.169391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.169368 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-wxc5w"] Nov 01 20:33:28.329423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.329325 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c"] Nov 01 20:33:28.335230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.335192 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-p8m9c"] Nov 01 20:33:28.467679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.467645 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5696e679-5d84-4352-8122-42262f1457e0" path="/var/lib/kubelet/pods/5696e679-5d84-4352-8122-42262f1457e0/volumes" Nov 01 20:33:28.468253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.468232 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" path="/var/lib/kubelet/pods/a2817334-701e-455f-9874-0ed3d1a2cc13/volumes" Nov 01 20:33:28.468792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.468772 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" path="/var/lib/kubelet/pods/f087cc7b-8d2d-4d9c-a057-36ad1f1863ed/volumes" Nov 01 20:33:28.469619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:28.469599 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" path="/var/lib/kubelet/pods/f21d6128-3b0a-4d41-828c-ed92bf4611a5/volumes" Nov 01 20:33:31.197517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:31.197475 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:31.915284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:31.915251 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:33:31.915704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:31.915687 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:33:34.227948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.227910 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz"] Nov 01 20:33:34.228615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228593 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" Nov 01 20:33:34.228689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228620 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" Nov 01 20:33:34.228689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228633 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" Nov 01 20:33:34.228689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228641 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" Nov 01 20:33:34.228689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228654 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" Nov 01 20:33:34.228689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228660 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" Nov 01 20:33:34.228689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228685 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3a53af7e-5492-4d2a-96d2-830f9387e5a4" containerName="revoke-keys" Nov 01 20:33:34.228873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228692 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a53af7e-5492-4d2a-96d2-830f9387e5a4" containerName="revoke-keys" Nov 01 20:33:34.228873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228724 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" Nov 01 20:33:34.228873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228730 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" Nov 01 20:33:34.228873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228750 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" Nov 01 20:33:34.228873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228756 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" Nov 01 20:33:34.229016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228904 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" Nov 01 20:33:34.229016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228918 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21d6128-3b0a-4d41-828c-ed92bf4611a5" containerName="istio-proxy" Nov 01 20:33:34.229016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228931 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2817334-701e-455f-9874-0ed3d1a2cc13" containerName="istio-proxy" Nov 01 20:33:34.229016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228940 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f087cc7b-8d2d-4d9c-a057-36ad1f1863ed" containerName="istio-proxy" Nov 01 20:33:34.229016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.228951 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a53af7e-5492-4d2a-96d2-830f9387e5a4" containerName="revoke-keys" Nov 01 20:33:34.232625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.232609 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.289033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.288976 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz"] Nov 01 20:33:34.331316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.331280 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkv8\" (UniqueName: \"kubernetes.io/projected/a812ddaf-b525-44b0-9f84-f421ee4f43f6-kube-api-access-kfkv8\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.331520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.331434 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.331561 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.331520 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.331817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.331786 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:33:34.331970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.331956 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:33:34.332028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.331978 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:33:34.333569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.333552 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:33:34.432859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.432821 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkv8\" (UniqueName: \"kubernetes.io/projected/a812ddaf-b525-44b0-9f84-f421ee4f43f6-kube-api-access-kfkv8\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.432859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.432902 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.433124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.432975 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.433674 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.433647 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.435462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.435441 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.489486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.489407 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkv8\" (UniqueName: \"kubernetes.io/projected/a812ddaf-b525-44b0-9f84-f421ee4f43f6-kube-api-access-kfkv8\") pod \"maas-api-revoke-sub-96383c53-k8trz\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.547244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.547206 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:34.924023 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:34.923992 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda812ddaf_b525_44b0_9f84_f421ee4f43f6.slice/crio-70564c2a74602246815bed577b21042f87e2f35e27b8164a580efaec1545853e WatchSource:0}: Error finding container 70564c2a74602246815bed577b21042f87e2f35e27b8164a580efaec1545853e: Status 404 returned error can't find the container with id 70564c2a74602246815bed577b21042f87e2f35e27b8164a580efaec1545853e Nov 01 20:33:34.925090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:34.925065 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz"] Nov 01 20:33:35.073807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:35.073767 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" event={"ID":"a812ddaf-b525-44b0-9f84-f421ee4f43f6","Type":"ContainerStarted","Data":"70564c2a74602246815bed577b21042f87e2f35e27b8164a580efaec1545853e"} Nov 01 20:33:35.891201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:35.891161 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:36.083636 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:36.083593 5633 generic.go:356] "Generic (PLEG): container finished" podID="a812ddaf-b525-44b0-9f84-f421ee4f43f6" containerID="06fd2621a3e1c9351816b46d30a26d3919b7c81a48eea25499f730d26a86dbdc" exitCode=0 Nov 01 20:33:36.083820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:36.083648 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" event={"ID":"a812ddaf-b525-44b0-9f84-f421ee4f43f6","Type":"ContainerDied","Data":"06fd2621a3e1c9351816b46d30a26d3919b7c81a48eea25499f730d26a86dbdc"} Nov 01 20:33:37.255531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.255499 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:37.373100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.373016 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-openshift-ca\") pod \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " Nov 01 20:33:37.373100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.373053 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a812ddaf-b525-44b0-9f84-f421ee4f43f6-kube-api-access-kfkv8\" (UniqueName: \"kubernetes.io/projected/a812ddaf-b525-44b0-9f84-f421ee4f43f6-kube-api-access-kfkv8\") pod \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " Nov 01 20:33:37.373100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.373076 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-xks-ca\") pod \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\" (UID: \"a812ddaf-b525-44b0-9f84-f421ee4f43f6\") " Nov 01 20:33:37.373432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.373410 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-openshift-ca" pod "a812ddaf-b525-44b0-9f84-f421ee4f43f6" (UID: "a812ddaf-b525-44b0-9f84-f421ee4f43f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:37.373613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.373591 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:37.375484 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.375452 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-xks-ca" pod "a812ddaf-b525-44b0-9f84-f421ee4f43f6" (UID: "a812ddaf-b525-44b0-9f84-f421ee4f43f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:37.375605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.375578 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a812ddaf-b525-44b0-9f84-f421ee4f43f6-kube-api-access-kfkv8" pod "a812ddaf-b525-44b0-9f84-f421ee4f43f6" (UID: "a812ddaf-b525-44b0-9f84-f421ee4f43f6"). InnerVolumeSpecName "kube-api-access-kfkv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:37.475150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.475112 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkv8\" (UniqueName: \"kubernetes.io/projected/a812ddaf-b525-44b0-9f84-f421ee4f43f6-kube-api-access-kfkv8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:37.475150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:37.475144 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a812ddaf-b525-44b0-9f84-f421ee4f43f6-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:38.108868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:38.108836 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" Nov 01 20:33:38.109091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:38.108864 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz" event={"ID":"a812ddaf-b525-44b0-9f84-f421ee4f43f6","Type":"ContainerDied","Data":"70564c2a74602246815bed577b21042f87e2f35e27b8164a580efaec1545853e"} Nov 01 20:33:38.109091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:38.108937 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70564c2a74602246815bed577b21042f87e2f35e27b8164a580efaec1545853e" Nov 01 20:33:40.630964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:40.630917 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:42.683192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:42.683153 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:33:44.124358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:44.085922 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8"] Nov 01 20:33:44.151657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:44.151620 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8184ce1-4dch8"] Nov 01 20:33:44.467379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:44.467316 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9057688e-8632-407f-beab-acf487a0b1d2" path="/var/lib/kubelet/pods/9057688e-8632-407f-beab-acf487a0b1d2/volumes" Nov 01 20:33:44.819441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:44.804288 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:45.912807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.912757 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl"] Nov 01 20:33:45.913449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.913431 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a812ddaf-b525-44b0-9f84-f421ee4f43f6" containerName="revoke-keys" Nov 01 20:33:45.913501 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.913451 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a812ddaf-b525-44b0-9f84-f421ee4f43f6" containerName="revoke-keys" Nov 01 20:33:45.913606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.913595 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a812ddaf-b525-44b0-9f84-f421ee4f43f6" containerName="revoke-keys" Nov 01 20:33:45.913640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.913610 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5696e679-5d84-4352-8122-42262f1457e0" containerName="istio-proxy" Nov 01 20:33:45.918369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.918352 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:45.920955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.920928 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:45.921037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.920985 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:45.921158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.921138 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktn7h\" (UniqueName: \"kubernetes.io/projected/650a9d50-e5ca-4c52-998b-1df06b5c139e-kube-api-access-ktn7h\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:45.970459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.970420 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:33:45.979155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.979082 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:33:45.979155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:45.979147 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:33:46.019427 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.019384 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:46.020453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.020429 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:33:46.022102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.022068 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:46.022194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.022146 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:46.022300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.022266 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktn7h\" (UniqueName: \"kubernetes.io/projected/650a9d50-e5ca-4c52-998b-1df06b5c139e-kube-api-access-ktn7h\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:46.023802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.023780 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:46.027279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.027250 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl"] Nov 01 20:33:46.027760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.027739 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:46.079882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.079849 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktn7h\" (UniqueName: \"kubernetes.io/projected/650a9d50-e5ca-4c52-998b-1df06b5c139e-kube-api-access-ktn7h\") pod \"maas-api-revoke-sub-28d198ed-bxrpl\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:46.159307 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:46.159266 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:33:46.232948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.232905 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:46.349310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.349277 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:46.679292 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:46.679263 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650a9d50_e5ca_4c52_998b_1df06b5c139e.slice/crio-5d1ae540db92ae2b3545e54b38637c04f399726d6382ea5342b2d65d9cd175f0 WatchSource:0}: Error finding container 5d1ae540db92ae2b3545e54b38637c04f399726d6382ea5342b2d65d9cd175f0: Status 404 returned error can't find the container with id 5d1ae540db92ae2b3545e54b38637c04f399726d6382ea5342b2d65d9cd175f0 Nov 01 20:33:46.681369 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.681324 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl"] Nov 01 20:33:46.716358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:46.716246 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:47.120323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.120282 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6"] Nov 01 20:33:47.169672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.169619 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-393dc263-hzrl6"] Nov 01 20:33:47.181294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.181257 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" event={"ID":"650a9d50-e5ca-4c52-998b-1df06b5c139e","Type":"ContainerStarted","Data":"5d1ae540db92ae2b3545e54b38637c04f399726d6382ea5342b2d65d9cd175f0"} Nov 01 20:33:47.568303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.568264 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f"] Nov 01 20:33:47.568916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.568883 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" containerID="cri-o://be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9" gracePeriod=30 Nov 01 20:33:47.580914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.580844 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:47.642296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.636466 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl"] Nov 01 20:33:47.642296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.636709 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" containerID="cri-o://13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0" gracePeriod=30 Nov 01 20:33:47.676351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.668679 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph"] Nov 01 20:33:47.676351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.669087 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" containerID="cri-o://90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8" gracePeriod=30 Nov 01 20:33:47.747618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.720579 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx"] Nov 01 20:33:47.747618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.720907 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" containerID="cri-o://88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16" gracePeriod=30 Nov 01 20:33:47.828291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.828206 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:47.952340 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:47.952255 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:48.078812 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.078719 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc"] Nov 01 20:33:48.083074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.083041 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-71335882-hskbc"] Nov 01 20:33:48.212557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.212521 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:48.214021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.213866 5633 generic.go:356] "Generic (PLEG): container finished" podID="650a9d50-e5ca-4c52-998b-1df06b5c139e" containerID="a8f0fbb4940e9074d79b617b8172525a33580817a0f5c223a210e1126eb41774" exitCode=0 Nov 01 20:33:48.214021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.213958 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" event={"ID":"650a9d50-e5ca-4c52-998b-1df06b5c139e","Type":"ContainerDied","Data":"a8f0fbb4940e9074d79b617b8172525a33580817a0f5c223a210e1126eb41774"} Nov 01 20:33:48.322280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.318432 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8"] Nov 01 20:33:48.322280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.318707 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" containerID="cri-o://18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87" gracePeriod=30 Nov 01 20:33:48.351877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.328492 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d"] Nov 01 20:33:48.351877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.328747 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" containerID="cri-o://fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653" gracePeriod=30 Nov 01 20:33:48.351877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.336464 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944"] Nov 01 20:33:48.351877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.336851 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" containerID="cri-o://9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c" gracePeriod=30 Nov 01 20:33:48.351877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.341153 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:48.471730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.471693 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a4437f-50d5-4c35-98df-a88fb0a57f6d" path="/var/lib/kubelet/pods/38a4437f-50d5-4c35-98df-a88fb0a57f6d/volumes" Nov 01 20:33:48.472278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.472261 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13cbea0-7683-4eef-9a2d-1331a07f94b8" path="/var/lib/kubelet/pods/d13cbea0-7683-4eef-9a2d-1331a07f94b8/volumes" Nov 01 20:33:48.740093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.740063 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568"] Nov 01 20:33:48.746140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.746092 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781346 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-envoy\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781387 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-data\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781422 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-socket\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781441 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-token\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781513 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-certs\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781659 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781648 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-credential-socket\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781713 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istiod-ca-cert\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781754 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-podinfo\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.781876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.781828 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp28w\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-kube-api-access-lp28w\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.813358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.812482 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2f37811a-openshift-default-dockercfg-qrt46\"" Nov 01 20:33:48.834888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.834844 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:48.837262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.837227 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568"] Nov 01 20:33:48.882767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.882735 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-envoy\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.882968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.882773 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-data\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.882968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.882824 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-socket\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.882968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.882847 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-token\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.882968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.882875 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-certs\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.882968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.882925 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-credential-socket\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.882996 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istiod-ca-cert\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.883040 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-podinfo\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.883107 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp28w\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-kube-api-access-lp28w\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.883236 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-data\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.883551 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-certs\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.883599 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-socket\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.883834 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-credential-socket\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.884997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.884835 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istiod-ca-cert\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.905377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.897619 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-envoy\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.930356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.925841 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-podinfo\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.940053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.939938 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp28w\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-kube-api-access-lp28w\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:48.940289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:48.940264 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-token\") pod \"e2e-derive-2f37811a-openshift-default-5cd79968f-l7568\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:49.012381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.012260 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:49.062131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.061984 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:49.392209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.392178 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:49.410977 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.410951 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-openshift-ca\") pod \"650a9d50-e5ca-4c52-998b-1df06b5c139e\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " Nov 01 20:33:49.411164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.411151 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650a9d50-e5ca-4c52-998b-1df06b5c139e-kube-api-access-ktn7h\" (UniqueName: \"kubernetes.io/projected/650a9d50-e5ca-4c52-998b-1df06b5c139e-kube-api-access-ktn7h\") pod \"650a9d50-e5ca-4c52-998b-1df06b5c139e\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " Nov 01 20:33:49.411227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.411207 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-xks-ca\") pod \"650a9d50-e5ca-4c52-998b-1df06b5c139e\" (UID: \"650a9d50-e5ca-4c52-998b-1df06b5c139e\") " Nov 01 20:33:49.411309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.411287 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-openshift-ca" pod "650a9d50-e5ca-4c52-998b-1df06b5c139e" (UID: "650a9d50-e5ca-4c52-998b-1df06b5c139e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:49.411740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.411714 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:49.413474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.413451 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650a9d50-e5ca-4c52-998b-1df06b5c139e-kube-api-access-ktn7h" pod "650a9d50-e5ca-4c52-998b-1df06b5c139e" (UID: "650a9d50-e5ca-4c52-998b-1df06b5c139e"). InnerVolumeSpecName "kube-api-access-ktn7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:49.413609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.413474 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-xks-ca" pod "650a9d50-e5ca-4c52-998b-1df06b5c139e" (UID: "650a9d50-e5ca-4c52-998b-1df06b5c139e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:49.499572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.499533 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m"] Nov 01 20:33:49.500275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.500254 5633 cpu_manager.go:414] "containerMap: removing container" podUID="650a9d50-e5ca-4c52-998b-1df06b5c139e" containerName="revoke-keys" Nov 01 20:33:49.500432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.500277 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650a9d50-e5ca-4c52-998b-1df06b5c139e" containerName="revoke-keys" Nov 01 20:33:49.500514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.500493 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="650a9d50-e5ca-4c52-998b-1df06b5c139e" containerName="revoke-keys" Nov 01 20:33:49.504485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.504459 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.513119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.513080 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktn7h\" (UniqueName: \"kubernetes.io/projected/650a9d50-e5ca-4c52-998b-1df06b5c139e-kube-api-access-ktn7h\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:49.513119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.513116 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650a9d50-e5ca-4c52-998b-1df06b5c139e-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:49.542967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.542923 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568"] Nov 01 20:33:49.568818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.564374 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:49.568818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.564481 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:49.568818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.564525 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:49.568818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.565184 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m"] Nov 01 20:33:49.614376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614341 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614499 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614530 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614551 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f85df6b-1a70-4df2-9c00-3240850c781d-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614630 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614668 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd925\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-kube-api-access-fd925\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614777 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614827 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.614969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.614943 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716259 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716396 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716433 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716538 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f85df6b-1a70-4df2-9c00-3240850c781d-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716587 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716622 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd925\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-kube-api-access-fd925\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716744 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716766 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716808 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716821 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.716991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.716888 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.717099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.717023 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.717264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.717236 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.717642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.717610 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f85df6b-1a70-4df2-9c00-3240850c781d-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.719442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.719416 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.720011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.719991 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.772881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.772849 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.773065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.773017 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd925\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-kube-api-access-fd925\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:49.831199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:49.831096 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:50.232074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.232041 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" Nov 01 20:33:50.232276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.232067 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl" event={"ID":"650a9d50-e5ca-4c52-998b-1df06b5c139e","Type":"ContainerDied","Data":"5d1ae540db92ae2b3545e54b38637c04f399726d6382ea5342b2d65d9cd175f0"} Nov 01 20:33:50.232276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.232130 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1ae540db92ae2b3545e54b38637c04f399726d6382ea5342b2d65d9cd175f0" Nov 01 20:33:50.234142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.234110 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" event={"ID":"0e5b127f-1db8-4a60-91c1-7bf056c693a3","Type":"ContainerStarted","Data":"bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1"} Nov 01 20:33:50.234303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.234168 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" event={"ID":"0e5b127f-1db8-4a60-91c1-7bf056c693a3","Type":"ContainerStarted","Data":"a0c0fc95728033fd9601de851e69013a50b96cfdade10e374d31a00d949927c4"} Nov 01 20:33:50.274005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.273975 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m"] Nov 01 20:33:50.274839 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:50.274815 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f85df6b_1a70_4df2_9c00_3240850c781d.slice/crio-2845236af244117af9ef3e9fbac64b1fbcccfa62bb39557a8df2284e8784001e WatchSource:0}: Error finding container 2845236af244117af9ef3e9fbac64b1fbcccfa62bb39557a8df2284e8784001e: Status 404 returned error can't find the container with id 2845236af244117af9ef3e9fbac64b1fbcccfa62bb39557a8df2284e8784001e Nov 01 20:33:50.278275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.278239 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:50.278422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.278312 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:50.278422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.278364 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:50.353482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.353419 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" podStartSLOduration=2.353400695 podStartE2EDuration="2.353400695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:50.353239068 +0000 UTC m=+1838.027498323" watchObservedRunningTime="2026-11-01 20:33:50.353400695 +0000 UTC m=+1838.027659944" Nov 01 20:33:50.416829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.416788 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:50.546068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.545966 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:50.584888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.584848 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:50.585074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.584919 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:50.632647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.632602 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-7rhph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:50.632814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:50.632674 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:51.062146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.062100 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:51.068621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.068584 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:51.246502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.246456 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" event={"ID":"8f85df6b-1a70-4df2-9c00-3240850c781d","Type":"ContainerStarted","Data":"0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399"} Nov 01 20:33:51.246711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.246526 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" event={"ID":"8f85df6b-1a70-4df2-9c00-3240850c781d","Type":"ContainerStarted","Data":"2845236af244117af9ef3e9fbac64b1fbcccfa62bb39557a8df2284e8784001e"} Nov 01 20:33:51.246791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.246750 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:51.247802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.247774 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:33:51.371581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.371472 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podStartSLOduration=2.3714561 podStartE2EDuration="2.3714561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:51.369811497 +0000 UTC m=+1839.044070779" watchObservedRunningTime="2026-11-01 20:33:51.3714561 +0000 UTC m=+1839.045715352" Nov 01 20:33:51.831563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.831525 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:51.832918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.832892 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 20:33:51.833052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.832950 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 20:33:51.868191 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.868152 5633 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-c259f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:51.868456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:51.868214 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:52.832612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:52.832066 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 20:33:52.832612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:52.832124 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 20:33:52.872279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:52.872193 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:53.138666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.138628 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:33:53.154736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.143155 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:33:53.176036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.175905 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-podinfo\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176036 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176032 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-data\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176155 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-socket\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176229 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-kube-api-access-vbzpt\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-kube-api-access-vbzpt\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176310 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-envoy\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176382 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-token\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-token\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176506 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-certs\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176562 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istiod-ca-cert\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.176759 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.176739 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-credential-socket\") pod \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\" (UID: \"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4\") " Nov 01 20:33:53.179533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.179499 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-credential-socket" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.180195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.180120 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-socket" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.180376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.180284 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-data" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.180469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.180394 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-certs" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.181021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.180992 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istiod-ca-cert" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.182722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.182683 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:53.231576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.230525 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-token" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.234479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.231912 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-kube-api-access-vbzpt" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "kube-api-access-vbzpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.234479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.233071 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-envoy" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.235088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.235051 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-podinfo" pod "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" (UID: "bf8de12b-5edf-4ebe-afbc-980a7d80aeb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.251608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.251316 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:33:53.253595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.253533 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:33:53.267313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.266885 5633 generic.go:356] "Generic (PLEG): container finished" podID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerID="90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8" exitCode=0 Nov 01 20:33:53.267313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.266971 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" event={"ID":"c57a2c36-19a0-404b-88f4-a453b3ce7513","Type":"ContainerDied","Data":"90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8"} Nov 01 20:33:53.267313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.267012 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" event={"ID":"c57a2c36-19a0-404b-88f4-a453b3ce7513","Type":"ContainerDied","Data":"96b19f80a1ad949f493534e5eeef59357fbd473dd0d479c7e9340f57f34e54f4"} Nov 01 20:33:53.267313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.267032 5633 scope.go:122] "RemoveContainer" containerID="90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8" Nov 01 20:33:53.267313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.267150 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph" Nov 01 20:33:53.268907 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.268879 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" Nov 01 20:33:53.269010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.268919 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" event={"ID":"f020231b-053e-410a-b714-dbc4baedf14b","Type":"ContainerDied","Data":"13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0"} Nov 01 20:33:53.269050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.269020 5633 generic.go:356] "Generic (PLEG): container finished" podID="f020231b-053e-410a-b714-dbc4baedf14b" containerID="13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0" exitCode=0 Nov 01 20:33:53.269187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.269171 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl" event={"ID":"f020231b-053e-410a-b714-dbc4baedf14b","Type":"ContainerDied","Data":"395bbbcbb939250718db6f55efe8cd2b33761dcae1ffbc1eb1c3e54719b55a38"} Nov 01 20:33:53.272515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.272468 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" Nov 01 20:33:53.272649 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.272563 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" event={"ID":"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4","Type":"ContainerDied","Data":"be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9"} Nov 01 20:33:53.273511 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.273461 5633 generic.go:356] "Generic (PLEG): container finished" podID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerID="be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9" exitCode=0 Nov 01 20:33:53.273622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.273598 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f" event={"ID":"bf8de12b-5edf-4ebe-afbc-980a7d80aeb4","Type":"ContainerDied","Data":"6a2e56e5af05f346d8de6cdd5c988f7f72bff032809bd3c2ef44aa4fa83f107a"} Nov 01 20:33:53.276769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.276741 5633 generic.go:356] "Generic (PLEG): container finished" podID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerID="88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16" exitCode=0 Nov 01 20:33:53.277103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.276858 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" event={"ID":"7882b493-231f-4c6d-a84d-cf30e24a6056","Type":"ContainerDied","Data":"88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16"} Nov 01 20:33:53.277103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.276969 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" event={"ID":"7882b493-231f-4c6d-a84d-cf30e24a6056","Type":"ContainerDied","Data":"459b7460838218258dd23ae1c1cd82ec9c34e7083e3ec00268a98c2080503dce"} Nov 01 20:33:53.277264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.277206 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx" Nov 01 20:33:53.280497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.279936 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-data\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.280497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280292 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-data" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.280497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280439 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-kube-api-access-sw4kr\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-kube-api-access-sw4kr\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.280497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280552 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-credential-socket\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.281799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280622 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-socket\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.281799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280686 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-podinfo\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.281799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280812 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-credential-socket" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.281799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280872 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-envoy\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.281799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280909 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57a2c36-19a0-404b-88f4-a453b3ce7513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a2c36-19a0-404b-88f4-a453b3ce7513-istiod-ca-cert\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.281799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.280964 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-token\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-token\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.281799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281029 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-certs\") pod \"c57a2c36-19a0-404b-88f4-a453b3ce7513\" (UID: \"c57a2c36-19a0-404b-88f4-a453b3ce7513\") " Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281842 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281864 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281881 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281895 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbzpt\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-kube-api-access-vbzpt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281909 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281922 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281936 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281948 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281956 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281964 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.281973 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.282194 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-certs" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.282225 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57a2c36-19a0-404b-88f4-a453b3ce7513-istiod-ca-cert" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.283020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.282291 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-socket" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.283880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.283832 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-envoy" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.284072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.284057 5633 scope.go:122] "RemoveContainer" containerID="90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8" Nov 01 20:33:53.284540 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:53.284461 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8\": container with ID starting with 90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8 not found: ID does not exist" containerID="90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8" Nov 01 20:33:53.284667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.284548 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8"} err="failed to get container status \"90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8\": rpc error: code = NotFound desc = could not find container \"90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8\": container with ID starting with 90974c1f3b1f6abc2408b2c6007122cdb195eda23fec43094a63ff1f43aedbd8 not found: ID does not exist" Nov 01 20:33:53.284667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.284567 5633 scope.go:122] "RemoveContainer" containerID="13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0" Nov 01 20:33:53.326062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.326017 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-podinfo" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.341356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.338720 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-token" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.341356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.338808 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-kube-api-access-sw4kr" pod "c57a2c36-19a0-404b-88f4-a453b3ce7513" (UID: "c57a2c36-19a0-404b-88f4-a453b3ce7513"). InnerVolumeSpecName "kube-api-access-sw4kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.341356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.339068 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383399 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-data\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383448 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-credential-socket\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383514 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7882b493-231f-4c6d-a84d-cf30e24a6056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882b493-231f-4c6d-a84d-cf30e24a6056-istiod-ca-cert\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383550 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-envoy\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383610 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-credential-socket\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383636 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-socket\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383673 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f020231b-053e-410a-b714-dbc4baedf14b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f020231b-053e-410a-b714-dbc4baedf14b-istiod-ca-cert\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383719 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-certs\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383751 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-podinfo\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383781 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-kube-api-access-48jb9\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-kube-api-access-48jb9\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383812 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-kube-api-access-jz927\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-kube-api-access-jz927\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383861 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-socket\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383897 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-token\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-token\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.384441 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f020231b-053e-410a-b714-dbc4baedf14b-istiod-ca-cert" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.384623 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-certs" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.385395 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-credential-socket" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.389703 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-credential-socket" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.395561 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-socket" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.396252 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-socket" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.383958 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-certs\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.403530 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-istio-token\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-istio-token\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.403591 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-envoy\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.403663 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f020231b-053e-410a-b714-dbc4baedf14b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f020231b-053e-410a-b714-dbc4baedf14b-istio-podinfo\") pod \"f020231b-053e-410a-b714-dbc4baedf14b\" (UID: \"f020231b-053e-410a-b714-dbc4baedf14b\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.403742 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-data\") pod \"7882b493-231f-4c6d-a84d-cf30e24a6056\" (UID: \"7882b493-231f-4c6d-a84d-cf30e24a6056\") " Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404667 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404692 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404710 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404727 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f020231b-053e-410a-b714-dbc4baedf14b-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404742 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw4kr\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-kube-api-access-sw4kr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404758 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404772 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404788 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404803 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404817 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404831 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a2c36-19a0-404b-88f4-a453b3ce7513-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404845 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a2c36-19a0-404b-88f4-a453b3ce7513-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.406703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.404861 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a2c36-19a0-404b-88f4-a453b3ce7513-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.426193 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.423750 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-certs" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.426193 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.424714 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-data" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.426193 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.425065 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7882b493-231f-4c6d-a84d-cf30e24a6056-istiod-ca-cert" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.426193 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.425129 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-data" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.427680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.427647 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f020231b-053e-410a-b714-dbc4baedf14b-istio-podinfo" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.429648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.429617 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-token" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.429648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.429626 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-envoy" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.429957 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.429862 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-istio-token" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.430253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.430215 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-kube-api-access-jz927" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "kube-api-access-jz927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.430561 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.430538 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-podinfo" pod "7882b493-231f-4c6d-a84d-cf30e24a6056" (UID: "7882b493-231f-4c6d-a84d-cf30e24a6056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.430839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.430823 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-kube-api-access-48jb9" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "kube-api-access-48jb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.431426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.431408 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-envoy" pod "f020231b-053e-410a-b714-dbc4baedf14b" (UID: "f020231b-053e-410a-b714-dbc4baedf14b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516453 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882b493-231f-4c6d-a84d-cf30e24a6056-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516503 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516522 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516540 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jb9\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-kube-api-access-48jb9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516560 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz927\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-kube-api-access-jz927\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516577 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516598 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516616 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f020231b-053e-410a-b714-dbc4baedf14b-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516632 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516648 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f020231b-053e-410a-b714-dbc4baedf14b-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516665 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882b493-231f-4c6d-a84d-cf30e24a6056-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.520771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.516683 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f020231b-053e-410a-b714-dbc4baedf14b-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.526551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.526523 5633 scope.go:122] "RemoveContainer" containerID="13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0" Nov 01 20:33:53.527379 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:53.526999 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0\": container with ID starting with 13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0 not found: ID does not exist" containerID="13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0" Nov 01 20:33:53.527379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.527044 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0"} err="failed to get container status \"13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0\": rpc error: code = NotFound desc = could not find container \"13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0\": container with ID starting with 13ad4bf7b2407df6e2c862850b04f1c0915f9b8534ed9b251914534558af4cc0 not found: ID does not exist" Nov 01 20:33:53.527379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.527064 5633 scope.go:122] "RemoveContainer" containerID="be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9" Nov 01 20:33:53.543112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.543011 5633 scope.go:122] "RemoveContainer" containerID="982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891" Nov 01 20:33:53.564629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.564509 5633 scope.go:122] "RemoveContainer" containerID="be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9" Nov 01 20:33:53.567219 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:53.564974 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9\": container with ID starting with be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9 not found: ID does not exist" containerID="be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9" Nov 01 20:33:53.567219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.565020 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9"} err="failed to get container status \"be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9\": rpc error: code = NotFound desc = could not find container \"be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9\": container with ID starting with be1ebc9816df8fcd9f93608f4e86e3aeedb3eb1f9c8f070a575de5043bb5d0a9 not found: ID does not exist" Nov 01 20:33:53.567219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.565047 5633 scope.go:122] "RemoveContainer" containerID="982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891" Nov 01 20:33:53.567219 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:53.565402 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891\": container with ID starting with 982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891 not found: ID does not exist" containerID="982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891" Nov 01 20:33:53.567219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.565431 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891"} err="failed to get container status \"982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891\": rpc error: code = NotFound desc = could not find container \"982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891\": container with ID starting with 982409be84d88113ecde2b5374eaed121d71ba47bb392fe6e8eeacbffaf1b891 not found: ID does not exist" Nov 01 20:33:53.567219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.565455 5633 scope.go:122] "RemoveContainer" containerID="88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16" Nov 01 20:33:53.625419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.625392 5633 scope.go:122] "RemoveContainer" containerID="88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16" Nov 01 20:33:53.625837 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:53.625805 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16\": container with ID starting with 88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16 not found: ID does not exist" containerID="88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16" Nov 01 20:33:53.625966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.625871 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16"} err="failed to get container status \"88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16\": rpc error: code = NotFound desc = could not find container \"88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16\": container with ID starting with 88e8116d5bddb22eae4fe8c0915130f604d899ba1e39e3f640a814bc6406bd16 not found: ID does not exist" Nov 01 20:33:53.833112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.832818 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 20:33:53.833112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.832895 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 20:33:53.847693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.847662 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:33:53.929822 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.929578 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:33:53.931884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.931812 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:33:53.937037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.936940 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-credential-socket\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937162 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9411d425-6b52-467d-819f-af21d34e68b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411d425-6b52-467d-819f-af21d34e68b6-istiod-ca-cert\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937268 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-kube-api-access-nz79h\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-kube-api-access-nz79h\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937347 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-socket\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937397 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9411d425-6b52-467d-819f-af21d34e68b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411d425-6b52-467d-819f-af21d34e68b6-istio-podinfo\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937429 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-istio-token\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-istio-token\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937522 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-certs\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937594 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-envoy\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937687 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-socket" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.937813 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.937694 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-data\") pod \"9411d425-6b52-467d-819f-af21d34e68b6\" (UID: \"9411d425-6b52-467d-819f-af21d34e68b6\") " Nov 01 20:33:53.938818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.938784 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-data" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.938979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.938961 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-certs" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.939429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.939401 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-credential-socket" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.941753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.941722 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9411d425-6b52-467d-819f-af21d34e68b6-istiod-ca-cert" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.942370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.942283 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.942370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.942314 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.942598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.942533 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.942598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.942559 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.942598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.942576 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411d425-6b52-467d-819f-af21d34e68b6-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.943794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.943435 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9411d425-6b52-467d-819f-af21d34e68b6-istio-podinfo" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.945317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.945271 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-kube-api-access-nz79h" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "kube-api-access-nz79h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.947787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.947752 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-envoy" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.947987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:53.947960 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-istio-token" pod "9411d425-6b52-467d-819f-af21d34e68b6" (UID: "9411d425-6b52-467d-819f-af21d34e68b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.043199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.042914 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-data\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043023 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-token\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-token\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043071 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-token\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-token\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043108 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-envoy\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043145 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-certs\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043175 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-socket\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043195 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-podinfo\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043219 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-kube-api-access-7rtb8\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-kube-api-access-7rtb8\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043242 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-credential-socket\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043273 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-envoy\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043294 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istiod-ca-cert\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043370 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-podinfo\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043412 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-certs\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043412 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-data" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043460 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-data\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043504 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-socket\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043538 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b653b1-2d99-4d23-a827-54e4cd53e659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b653b1-2d99-4d23-a827-54e4cd53e659-istiod-ca-cert\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043645 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-kube-api-access-pdlwk\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-kube-api-access-pdlwk\") pod \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\" (UID: \"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d\") " Nov 01 20:33:54.043797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043677 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-credential-socket\") pod \"81b653b1-2d99-4d23-a827-54e4cd53e659\" (UID: \"81b653b1-2d99-4d23-a827-54e4cd53e659\") " Nov 01 20:33:54.044411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.043952 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-credential-socket" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.045014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.044790 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-socket" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.045259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.045238 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b653b1-2d99-4d23-a827-54e4cd53e659-istiod-ca-cert" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:54.045745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.045711 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-socket" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.045880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.045763 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-certs" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.045880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.045820 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-data" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.046104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.046083 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istiod-ca-cert" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:54.046439 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.046383 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-credential-socket" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.046774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.046749 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-certs" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.047619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.047588 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-token" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.047828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.047788 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-token" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.048290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048245 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048278 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048291 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048306 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048321 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048364 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048377 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048391 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz79h\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-kube-api-access-nz79h\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048414 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048427 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411d425-6b52-467d-819f-af21d34e68b6-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048441 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411d425-6b52-467d-819f-af21d34e68b6-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048452 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048465 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048477 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b653b1-2d99-4d23-a827-54e4cd53e659-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048490 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411d425-6b52-467d-819f-af21d34e68b6-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.048747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.048504 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.050013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.049946 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-envoy" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.050832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.050766 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-kube-api-access-pdlwk" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "kube-api-access-pdlwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.050930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.050886 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-kube-api-access-7rtb8" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "kube-api-access-7rtb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.051702 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.051674 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-envoy" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.054022 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.053985 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-podinfo" pod "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" (UID: "971a1bdc-27ec-48fa-afb7-dea6c5b7b02d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:54.055409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.054999 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-podinfo" pod "81b653b1-2d99-4d23-a827-54e4cd53e659" (UID: "81b653b1-2d99-4d23-a827-54e4cd53e659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:54.120891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.120855 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f"] Nov 01 20:33:54.125642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.125603 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-c259f"] Nov 01 20:33:54.149687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.149651 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlwk\" (UniqueName: \"kubernetes.io/projected/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-kube-api-access-pdlwk\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.149687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.149681 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.149687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.149691 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b653b1-2d99-4d23-a827-54e4cd53e659-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.149942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.149701 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtb8\" (UniqueName: \"kubernetes.io/projected/81b653b1-2d99-4d23-a827-54e4cd53e659-kube-api-access-7rtb8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.149942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.149711 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.149942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.149719 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.228784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.228746 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl"] Nov 01 20:33:54.232733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.232708 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-hg8xl"] Nov 01 20:33:54.312132 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.312042 5633 generic.go:356] "Generic (PLEG): container finished" podID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerID="f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735" exitCode=0 Nov 01 20:33:54.312307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.312121 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" event={"ID":"d24418ce-cbc8-414a-92f7-9f2ed7f338b3","Type":"ContainerDied","Data":"f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735"} Nov 01 20:33:54.312720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.312681 5633 scope.go:122] "RemoveContainer" containerID="f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735" Nov 01 20:33:54.314852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.314818 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:54.314975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.314915 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:54.314975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.314958 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:33:54.317042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.317018 5633 generic.go:356] "Generic (PLEG): container finished" podID="9411d425-6b52-467d-819f-af21d34e68b6" containerID="9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c" exitCode=0 Nov 01 20:33:54.317161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.317098 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" Nov 01 20:33:54.317245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.317101 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" event={"ID":"9411d425-6b52-467d-819f-af21d34e68b6","Type":"ContainerDied","Data":"9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c"} Nov 01 20:33:54.317245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.317228 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944" event={"ID":"9411d425-6b52-467d-819f-af21d34e68b6","Type":"ContainerDied","Data":"f711c6325fd68f138303f93641fe3cab5f725b14f2691ffa34e6edd7eaf54551"} Nov 01 20:33:54.317387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.317253 5633 scope.go:122] "RemoveContainer" containerID="9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c" Nov 01 20:33:54.318688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.318666 5633 generic.go:356] "Generic (PLEG): container finished" podID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerID="18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87" exitCode=0 Nov 01 20:33:54.318799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.318708 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" Nov 01 20:33:54.318799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.318767 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" event={"ID":"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d","Type":"ContainerDied","Data":"18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87"} Nov 01 20:33:54.318924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.318805 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8" event={"ID":"971a1bdc-27ec-48fa-afb7-dea6c5b7b02d","Type":"ContainerDied","Data":"f00db3eb1756ccc6b23a6be0f75060c19a873c27da197f77083ea58cf5f5f53e"} Nov 01 20:33:54.320900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.320875 5633 generic.go:356] "Generic (PLEG): container finished" podID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerID="fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653" exitCode=0 Nov 01 20:33:54.321074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.321057 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" event={"ID":"81b653b1-2d99-4d23-a827-54e4cd53e659","Type":"ContainerDied","Data":"fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653"} Nov 01 20:33:54.321150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.321103 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" event={"ID":"81b653b1-2d99-4d23-a827-54e4cd53e659","Type":"ContainerDied","Data":"850c70c94b97b9192f8c4284dd0d18c777840076701072299cdd1f730167e1eb"} Nov 01 20:33:54.321213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.321181 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d" Nov 01 20:33:54.330793 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.330772 5633 scope.go:122] "RemoveContainer" containerID="9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c" Nov 01 20:33:54.331120 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:54.331100 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c\": container with ID starting with 9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c not found: ID does not exist" containerID="9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c" Nov 01 20:33:54.331188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.331131 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c"} err="failed to get container status \"9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c\": rpc error: code = NotFound desc = could not find container \"9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c\": container with ID starting with 9be68201b70afceb9d0a51f7a3ee239fb3b9c24addf65663a6ce17659cbb0f5c not found: ID does not exist" Nov 01 20:33:54.331188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.331152 5633 scope.go:122] "RemoveContainer" containerID="18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87" Nov 01 20:33:54.343084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.343061 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx"] Nov 01 20:33:54.344671 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.344653 5633 scope.go:122] "RemoveContainer" containerID="18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87" Nov 01 20:33:54.345187 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:54.345160 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87\": container with ID starting with 18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87 not found: ID does not exist" containerID="18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87" Nov 01 20:33:54.345260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.345200 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87"} err="failed to get container status \"18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87\": rpc error: code = NotFound desc = could not find container \"18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87\": container with ID starting with 18be5229d04d23eff7e73cf4830def21b8d4ad2174ac18eadee125352ac8fc87 not found: ID does not exist" Nov 01 20:33:54.345260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.345226 5633 scope.go:122] "RemoveContainer" containerID="fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653" Nov 01 20:33:54.347980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.347955 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-z2nvx"] Nov 01 20:33:54.362488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.362452 5633 scope.go:122] "RemoveContainer" containerID="972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920" Nov 01 20:33:54.414163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.414120 5633 scope.go:122] "RemoveContainer" containerID="fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653" Nov 01 20:33:54.420192 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:54.420150 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653\": container with ID starting with fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653 not found: ID does not exist" containerID="fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653" Nov 01 20:33:54.420411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.420198 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653"} err="failed to get container status \"fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653\": rpc error: code = NotFound desc = could not find container \"fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653\": container with ID starting with fe958c944ee5c2fecc380a865831961b5a6e96788a88934504f3025ec0239653 not found: ID does not exist" Nov 01 20:33:54.420411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.420226 5633 scope.go:122] "RemoveContainer" containerID="972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920" Nov 01 20:33:54.420994 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:33:54.420962 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920\": container with ID starting with 972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920 not found: ID does not exist" containerID="972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920" Nov 01 20:33:54.421091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.421005 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920"} err="failed to get container status \"972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920\": rpc error: code = NotFound desc = could not find container \"972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920\": container with ID starting with 972c58a27fe2451e44792ebc3051c324fccea1fe7cfbf14e2498a2e71beba920 not found: ID does not exist" Nov 01 20:33:54.470545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.470510 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" path="/var/lib/kubelet/pods/7882b493-231f-4c6d-a84d-cf30e24a6056/volumes" Nov 01 20:33:54.471055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.471040 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" path="/var/lib/kubelet/pods/bf8de12b-5edf-4ebe-afbc-980a7d80aeb4/volumes" Nov 01 20:33:54.471522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.471507 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f020231b-053e-410a-b714-dbc4baedf14b" path="/var/lib/kubelet/pods/f020231b-053e-410a-b714-dbc4baedf14b/volumes" Nov 01 20:33:54.478533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.478508 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph"] Nov 01 20:33:54.483271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.483245 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-7rhph"] Nov 01 20:33:54.674496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.674446 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d"] Nov 01 20:33:54.682718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.682679 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-mg58d"] Nov 01 20:33:54.771863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.771824 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944"] Nov 01 20:33:54.776449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.776415 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-tc944"] Nov 01 20:33:54.831915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.831586 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 20:33:54.831915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.831635 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 20:33:54.961409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.961369 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8"] Nov 01 20:33:54.968820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:54.968782 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-twxc8"] Nov 01 20:33:55.256442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.255625 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk"] Nov 01 20:33:55.256711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256687 5633 cpu_manager.go:414] "containerMap: removing container" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" Nov 01 20:33:55.256777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256714 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" Nov 01 20:33:55.256777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256741 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" Nov 01 20:33:55.256777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256750 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" Nov 01 20:33:55.256777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256766 5633 cpu_manager.go:414] "containerMap: removing container" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" Nov 01 20:33:55.256777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256775 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256799 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256823 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256859 5633 cpu_manager.go:414] "containerMap: removing container" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256867 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256911 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256919 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256931 5633 cpu_manager.go:414] "containerMap: removing container" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256938 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256949 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" Nov 01 20:33:55.257037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.256957 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257106 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f020231b-053e-410a-b714-dbc4baedf14b" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257116 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257127 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257140 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257151 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257161 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7882b493-231f-4c6d-a84d-cf30e24a6056" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257172 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257185 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9411d425-6b52-467d-819f-af21d34e68b6" containerName="istio-proxy" Nov 01 20:33:55.257594 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.257197 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" Nov 01 20:33:55.263898 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.263865 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.267509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.267481 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.267685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.267518 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbh9\" (UniqueName: \"kubernetes.io/projected/d9d3128f-9731-4ec4-8045-36461283f7b9-kube-api-access-gzbh9\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.267685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.267548 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.273370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.272931 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk"] Nov 01 20:33:55.349390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.346973 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:33:55.349390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.347065 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:33:55.349390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.347150 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:33:55.349390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.347194 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:33:55.363262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.363215 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" event={"ID":"d24418ce-cbc8-414a-92f7-9f2ed7f338b3","Type":"ContainerStarted","Data":"9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0"} Nov 01 20:33:55.368602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.368569 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.368923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.368899 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbh9\" (UniqueName: \"kubernetes.io/projected/d9d3128f-9731-4ec4-8045-36461283f7b9-kube-api-access-gzbh9\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.369045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.368962 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.370194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.370166 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.376526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.376426 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.442472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.442439 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbh9\" (UniqueName: \"kubernetes.io/projected/d9d3128f-9731-4ec4-8045-36461283f7b9-kube-api-access-gzbh9\") pod \"maas-api-revoke-sub-ac2da8f3-xkfwk\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.653272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.653177 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:33:55.838311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.837821 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 20:33:55.838311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:55.837894 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 20:33:56.083047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.083007 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk"] Nov 01 20:33:56.209026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.208940 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:56.229358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.209633 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:33:56.229815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.229704 5633 patch_prober.go:29] interesting pod/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 20:33:56.230014 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.229989 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 20:33:56.380123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.379487 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:56.384504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.384478 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" event={"ID":"d9d3128f-9731-4ec4-8045-36461283f7b9","Type":"ContainerStarted","Data":"53e08b08998ea62c149401b64e4472b93c2b8a94353fe095c7457ea123c6a99c"} Nov 01 20:33:56.468551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.468516 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b653b1-2d99-4d23-a827-54e4cd53e659" path="/var/lib/kubelet/pods/81b653b1-2d99-4d23-a827-54e4cd53e659/volumes" Nov 01 20:33:56.469077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.469063 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9411d425-6b52-467d-819f-af21d34e68b6" path="/var/lib/kubelet/pods/9411d425-6b52-467d-819f-af21d34e68b6/volumes" Nov 01 20:33:56.469530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.469516 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971a1bdc-27ec-48fa-afb7-dea6c5b7b02d" path="/var/lib/kubelet/pods/971a1bdc-27ec-48fa-afb7-dea6c5b7b02d/volumes" Nov 01 20:33:56.470307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.470291 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57a2c36-19a0-404b-88f4-a453b3ce7513" path="/var/lib/kubelet/pods/c57a2c36-19a0-404b-88f4-a453b3ce7513/volumes" Nov 01 20:33:56.775358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.770645 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2"] Nov 01 20:33:56.800357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.799940 5633 cpu_manager.go:414] "containerMap: removing container" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" Nov 01 20:33:56.800357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.799974 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8de12b-5edf-4ebe-afbc-980a7d80aeb4" containerName="istio-proxy" Nov 01 20:33:56.809356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.805653 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:56.838184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.825961 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2"] Nov 01 20:33:56.838184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.837683 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 20:33:56.838184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.837745 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 20:33:56.863353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.861917 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2f37811a\"" Nov 01 20:33:56.863353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.861916 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2f37811a-dockercfg-tkn96\"" Nov 01 20:33:56.919260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.919219 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6ff52c-323d-47eb-af80-0160daf45272-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:56.920411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.920373 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7x79\" (UniqueName: \"kubernetes.io/projected/2d6ff52c-323d-47eb-af80-0160daf45272-kube-api-access-c7x79\") pod \"payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:56.927864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.927833 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs"] Nov 01 20:33:56.933285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.933261 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:56.950257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.950223 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv"] Nov 01 20:33:56.954856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.954777 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:56.994192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:56.994153 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs"] Nov 01 20:33:57.029349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.029291 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235939a2-8244-4b64-a584-2fc16f2ee829-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:57.029567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.029388 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2fc\" (UniqueName: \"kubernetes.io/projected/0407010c-1b6e-4166-9854-a7502c229b34-kube-api-access-qh2fc\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.029567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.029519 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.029680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.029602 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tm9p\" (UniqueName: \"kubernetes.io/projected/235939a2-8244-4b64-a584-2fc16f2ee829-kube-api-access-8tm9p\") pod \"payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:57.029739 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.029699 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6ff52c-323d-47eb-af80-0160daf45272-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:57.029739 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.029727 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7x79\" (UniqueName: \"kubernetes.io/projected/2d6ff52c-323d-47eb-af80-0160daf45272-kube-api-access-c7x79\") pod \"payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:57.029848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.029752 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.030787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.030757 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6ff52c-323d-47eb-af80-0160daf45272-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:57.041721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.041682 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv"] Nov 01 20:33:57.129411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.128853 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7x79\" (UniqueName: \"kubernetes.io/projected/2d6ff52c-323d-47eb-af80-0160daf45272-kube-api-access-c7x79\") pod \"payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:57.130992 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.130968 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.131090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.131028 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tm9p\" (UniqueName: \"kubernetes.io/projected/235939a2-8244-4b64-a584-2fc16f2ee829-kube-api-access-8tm9p\") pod \"payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:57.131090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.131076 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.131204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.131119 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235939a2-8244-4b64-a584-2fc16f2ee829-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:57.131204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.131146 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2fc\" (UniqueName: \"kubernetes.io/projected/0407010c-1b6e-4166-9854-a7502c229b34-kube-api-access-qh2fc\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.132476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.131948 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.133137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.133113 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235939a2-8244-4b64-a584-2fc16f2ee829-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:57.134621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.134597 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.149385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.149296 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:57.204285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.203504 5633 patch_prober.go:29] interesting pod/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 20:33:57.204285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.203571 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 20:33:57.211418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.205425 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tm9p\" (UniqueName: \"kubernetes.io/projected/235939a2-8244-4b64-a584-2fc16f2ee829-kube-api-access-8tm9p\") pod \"payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:57.237681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.237582 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2fc\" (UniqueName: \"kubernetes.io/projected/0407010c-1b6e-4166-9854-a7502c229b34-kube-api-access-qh2fc\") pod \"maas-api-revoke-sub-613cb867-2wczs\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.249169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.249137 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:33:57.273859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.273624 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:57.336342 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.335040 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:33:57.422173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.422068 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" event={"ID":"d9d3128f-9731-4ec4-8045-36461283f7b9","Type":"ContainerStarted","Data":"182c35002786ec3296ae7976eb37ab5be3e8f2bc9a675772dbc67aff3e199f7e"} Nov 01 20:33:57.539211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.539141 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" podStartSLOduration=2.539119291 podStartE2EDuration="2.539119291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:57.536732353 +0000 UTC m=+1845.210991605" watchObservedRunningTime="2026-11-01 20:33:57.539119291 +0000 UTC m=+1845.213378546" Nov 01 20:33:57.579959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.578943 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2"] Nov 01 20:33:57.579959 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:57.579236 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6ff52c_323d_47eb_af80_0160daf45272.slice/crio-0922b7d3c92dbeb845c5792d99f1fb31f797d6f7e1a42418a1ca70b65258049b WatchSource:0}: Error finding container 0922b7d3c92dbeb845c5792d99f1fb31f797d6f7e1a42418a1ca70b65258049b: Status 404 returned error can't find the container with id 0922b7d3c92dbeb845c5792d99f1fb31f797d6f7e1a42418a1ca70b65258049b Nov 01 20:33:57.729768 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:57.729738 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0407010c_1b6e_4166_9854_a7502c229b34.slice/crio-14591638b88c0ac6d463eb5aad0ba90257adb6be40622022bd0debe8f114b559 WatchSource:0}: Error finding container 14591638b88c0ac6d463eb5aad0ba90257adb6be40622022bd0debe8f114b559: Status 404 returned error can't find the container with id 14591638b88c0ac6d463eb5aad0ba90257adb6be40622022bd0debe8f114b559 Nov 01 20:33:57.731581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.731556 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs"] Nov 01 20:33:57.769416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.768622 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv"] Nov 01 20:33:57.769744 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:33:57.769713 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235939a2_8244_4b64_a584_2fc16f2ee829.slice/crio-b06d4b5aec7bfb19cb1bbec1e3af448b141da6ca5e62929ff2e41680fd880a0e WatchSource:0}: Error finding container b06d4b5aec7bfb19cb1bbec1e3af448b141da6ca5e62929ff2e41680fd880a0e: Status 404 returned error can't find the container with id b06d4b5aec7bfb19cb1bbec1e3af448b141da6ca5e62929ff2e41680fd880a0e Nov 01 20:33:57.836446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.836415 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:57.837003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.836970 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:57.838231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:57.838207 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:33:58.208873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.203138 5633 patch_prober.go:29] interesting pod/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 20:33:58.208873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.203196 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 20:33:58.429299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.429209 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" event={"ID":"2d6ff52c-323d-47eb-af80-0160daf45272","Type":"ContainerStarted","Data":"c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606"} Nov 01 20:33:58.429299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.429265 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" event={"ID":"2d6ff52c-323d-47eb-af80-0160daf45272","Type":"ContainerStarted","Data":"0922b7d3c92dbeb845c5792d99f1fb31f797d6f7e1a42418a1ca70b65258049b"} Nov 01 20:33:58.429598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.429318 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:33:58.431160 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.431131 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" event={"ID":"235939a2-8244-4b64-a584-2fc16f2ee829","Type":"ContainerStarted","Data":"a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97"} Nov 01 20:33:58.431314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.431182 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" event={"ID":"235939a2-8244-4b64-a584-2fc16f2ee829","Type":"ContainerStarted","Data":"b06d4b5aec7bfb19cb1bbec1e3af448b141da6ca5e62929ff2e41680fd880a0e"} Nov 01 20:33:58.431314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.431212 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:33:58.432983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.432952 5633 generic.go:356] "Generic (PLEG): container finished" podID="d9d3128f-9731-4ec4-8045-36461283f7b9" containerID="182c35002786ec3296ae7976eb37ab5be3e8f2bc9a675772dbc67aff3e199f7e" exitCode=0 Nov 01 20:33:58.433087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.433008 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" event={"ID":"d9d3128f-9731-4ec4-8045-36461283f7b9","Type":"ContainerDied","Data":"182c35002786ec3296ae7976eb37ab5be3e8f2bc9a675772dbc67aff3e199f7e"} Nov 01 20:33:58.434643 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.434623 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" event={"ID":"0407010c-1b6e-4166-9854-a7502c229b34","Type":"ContainerStarted","Data":"6a25274d6e8b71d57a80e643fb6b3319a514c5a8387968200862087a48349a0f"} Nov 01 20:33:58.434748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.434656 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" event={"ID":"0407010c-1b6e-4166-9854-a7502c229b34","Type":"ContainerStarted","Data":"14591638b88c0ac6d463eb5aad0ba90257adb6be40622022bd0debe8f114b559"} Nov 01 20:33:58.520188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.520154 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:33:58.543617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.543549 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" podStartSLOduration=2.5435301299999997 podStartE2EDuration="2.54353013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:58.542968609 +0000 UTC m=+1846.217227935" watchObservedRunningTime="2026-11-01 20:33:58.54353013 +0000 UTC m=+1846.217789386" Nov 01 20:33:58.777059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.777003 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" podStartSLOduration=2.776989049 podStartE2EDuration="2.776989049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:58.661965973 +0000 UTC m=+1846.336225230" watchObservedRunningTime="2026-11-01 20:33:58.776989049 +0000 UTC m=+1846.451248620" Nov 01 20:33:58.777266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:58.777144 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" podStartSLOduration=2.777138275 podStartE2EDuration="2.777138275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:58.775185234 +0000 UTC m=+1846.449444489" watchObservedRunningTime="2026-11-01 20:33:58.777138275 +0000 UTC m=+1846.451397587" Nov 01 20:33:59.205418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:59.195781 5633 patch_prober.go:29] interesting pod/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 20:33:59.205418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:59.195844 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 20:33:59.443250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:59.443213 5633 generic.go:356] "Generic (PLEG): container finished" podID="0407010c-1b6e-4166-9854-a7502c229b34" containerID="6a25274d6e8b71d57a80e643fb6b3319a514c5a8387968200862087a48349a0f" exitCode=0 Nov 01 20:33:59.443761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:33:59.443305 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" event={"ID":"0407010c-1b6e-4166-9854-a7502c229b34","Type":"ContainerDied","Data":"6a25274d6e8b71d57a80e643fb6b3319a514c5a8387968200862087a48349a0f"} Nov 01 20:34:00.152390 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.148796 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:34:00.195375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.193965 5633 patch_prober.go:29] interesting pod/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 20:34:00.195375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.194050 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 20:34:00.353432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.352167 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-xks-ca\") pod \"d9d3128f-9731-4ec4-8045-36461283f7b9\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " Nov 01 20:34:00.353432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.352275 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d3128f-9731-4ec4-8045-36461283f7b9-kube-api-access-gzbh9\" (UniqueName: \"kubernetes.io/projected/d9d3128f-9731-4ec4-8045-36461283f7b9-kube-api-access-gzbh9\") pod \"d9d3128f-9731-4ec4-8045-36461283f7b9\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " Nov 01 20:34:00.353432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.352373 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-openshift-ca\") pod \"d9d3128f-9731-4ec4-8045-36461283f7b9\" (UID: \"d9d3128f-9731-4ec4-8045-36461283f7b9\") " Nov 01 20:34:00.387378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.369062 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-openshift-ca" pod "d9d3128f-9731-4ec4-8045-36461283f7b9" (UID: "d9d3128f-9731-4ec4-8045-36461283f7b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:34:00.387378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.378515 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-xks-ca" pod "d9d3128f-9731-4ec4-8045-36461283f7b9" (UID: "d9d3128f-9731-4ec4-8045-36461283f7b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:34:00.416370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.397387 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d3128f-9731-4ec4-8045-36461283f7b9-kube-api-access-gzbh9" pod "d9d3128f-9731-4ec4-8045-36461283f7b9" (UID: "d9d3128f-9731-4ec4-8045-36461283f7b9"). InnerVolumeSpecName "kube-api-access-gzbh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:00.498355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.497882 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:00.498355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.497918 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d3128f-9731-4ec4-8045-36461283f7b9-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:00.498355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.497933 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbh9\" (UniqueName: \"kubernetes.io/projected/d9d3128f-9731-4ec4-8045-36461283f7b9-kube-api-access-gzbh9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:00.644433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.644143 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" Nov 01 20:34:00.684385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.665195 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk" event={"ID":"d9d3128f-9731-4ec4-8045-36461283f7b9","Type":"ContainerDied","Data":"53e08b08998ea62c149401b64e4472b93c2b8a94353fe095c7457ea123c6a99c"} Nov 01 20:34:00.684385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.665267 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53e08b08998ea62c149401b64e4472b93c2b8a94353fe095c7457ea123c6a99c" Nov 01 20:34:00.958350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:00.957917 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:34:01.066907 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.066863 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-openshift-ca\") pod \"0407010c-1b6e-4166-9854-a7502c229b34\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " Nov 01 20:34:01.067096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.066957 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-xks-ca\") pod \"0407010c-1b6e-4166-9854-a7502c229b34\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " Nov 01 20:34:01.067096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.067042 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0407010c-1b6e-4166-9854-a7502c229b34-kube-api-access-qh2fc\" (UniqueName: \"kubernetes.io/projected/0407010c-1b6e-4166-9854-a7502c229b34-kube-api-access-qh2fc\") pod \"0407010c-1b6e-4166-9854-a7502c229b34\" (UID: \"0407010c-1b6e-4166-9854-a7502c229b34\") " Nov 01 20:34:01.067320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.067293 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-openshift-ca" pod "0407010c-1b6e-4166-9854-a7502c229b34" (UID: "0407010c-1b6e-4166-9854-a7502c229b34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:34:01.067607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.067587 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:01.070266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.069692 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0407010c-1b6e-4166-9854-a7502c229b34-kube-api-access-qh2fc" pod "0407010c-1b6e-4166-9854-a7502c229b34" (UID: "0407010c-1b6e-4166-9854-a7502c229b34"). InnerVolumeSpecName "kube-api-access-qh2fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:01.070788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.070759 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-xks-ca" pod "0407010c-1b6e-4166-9854-a7502c229b34" (UID: "0407010c-1b6e-4166-9854-a7502c229b34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:34:01.170503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.170065 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0407010c-1b6e-4166-9854-a7502c229b34-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:01.170503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.170115 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh2fc\" (UniqueName: \"kubernetes.io/projected/0407010c-1b6e-4166-9854-a7502c229b34-kube-api-access-qh2fc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:01.193034 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.192998 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:34:01.194307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.194278 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:34:01.653178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.653149 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" Nov 01 20:34:01.653647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.653188 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs" event={"ID":"0407010c-1b6e-4166-9854-a7502c229b34","Type":"ContainerDied","Data":"14591638b88c0ac6d463eb5aad0ba90257adb6be40622022bd0debe8f114b559"} Nov 01 20:34:01.653647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:01.653235 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14591638b88c0ac6d463eb5aad0ba90257adb6be40622022bd0debe8f114b559" Nov 01 20:34:03.387857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:03.387813 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:05.878055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:05.878006 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:06.057688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:06.057651 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:07.121675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:07.121636 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:07.257900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:07.257859 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:07.387708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:07.387613 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:07.518744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:07.518707 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:08.399322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:08.399259 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:09.002131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:09.002092 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:09.447120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:09.447043 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:34:09.447120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:09.447114 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:34:10.306771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.306737 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd"] Nov 01 20:34:10.307387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.307374 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d9d3128f-9731-4ec4-8045-36461283f7b9" containerName="revoke-keys" Nov 01 20:34:10.307441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.307388 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d3128f-9731-4ec4-8045-36461283f7b9" containerName="revoke-keys" Nov 01 20:34:10.307441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.307410 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0407010c-1b6e-4166-9854-a7502c229b34" containerName="revoke-keys" Nov 01 20:34:10.307441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.307416 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0407010c-1b6e-4166-9854-a7502c229b34" containerName="revoke-keys" Nov 01 20:34:10.307561 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.307526 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d3128f-9731-4ec4-8045-36461283f7b9" containerName="revoke-keys" Nov 01 20:34:10.307561 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.307537 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0407010c-1b6e-4166-9854-a7502c229b34" containerName="revoke-keys" Nov 01 20:34:10.311096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.311081 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.368096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.368048 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd"] Nov 01 20:34:10.401125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.401095 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:10.406858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.406733 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjrsv\" (UniqueName: \"kubernetes.io/projected/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-kube-api-access-fjrsv\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.406858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.406807 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-model-cache\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.407577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.407495 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tls-certs\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.407577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.407536 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-dshm\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.407825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.407800 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tmp-dir\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.407950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.407844 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-home\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.410519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.410494 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f154f7\"/\"explicit-model-f154f7-kserve-self-signed-certs\"" Nov 01 20:34:10.410633 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.410522 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f154f7\"/\"default-dockercfg-rqb25\"" Nov 01 20:34:10.413130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.413111 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f154f7\"/\"kube-root-ca.crt\"" Nov 01 20:34:10.413195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.413162 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f154f7\"/\"openshift-service-ca.crt\"" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.509297 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrsv\" (UniqueName: \"kubernetes.io/projected/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-kube-api-access-fjrsv\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.509379 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-model-cache\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.509561 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tls-certs\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.509591 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-dshm\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.509625 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tmp-dir\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.509657 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-home\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.509880 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-model-cache\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.510172 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-home\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.510318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.510241 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tmp-dir\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.514791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.512558 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tls-certs\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.516621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.516541 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-dshm\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.566507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.566433 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrsv\" (UniqueName: \"kubernetes.io/projected/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-kube-api-access-fjrsv\") pod \"explicit-model-f154f7-kserve-5b89475549-6ffnd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:10.625365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:10.625305 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:11.037064 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:34:11.037021 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828e7588_20ab_4b25_b265_4cfa5b4c7ccd.slice/crio-b6d20958865c7cd66d213a08e2d1954e6205978f9344470de47508dd857fa631 WatchSource:0}: Error finding container b6d20958865c7cd66d213a08e2d1954e6205978f9344470de47508dd857fa631: Status 404 returned error can't find the container with id b6d20958865c7cd66d213a08e2d1954e6205978f9344470de47508dd857fa631 Nov 01 20:34:11.037634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:11.037607 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd"] Nov 01 20:34:11.743423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:11.743381 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" event={"ID":"828e7588-20ab-4b25-b265-4cfa5b4c7ccd","Type":"ContainerStarted","Data":"e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf"} Nov 01 20:34:11.743821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:11.743433 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" event={"ID":"828e7588-20ab-4b25-b265-4cfa5b4c7ccd","Type":"ContainerStarted","Data":"b6d20958865c7cd66d213a08e2d1954e6205978f9344470de47508dd857fa631"} Nov 01 20:34:11.858278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:11.858225 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" podStartSLOduration=1.858209153 podStartE2EDuration="1.858209153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:11.856188515 +0000 UTC m=+1859.530447771" watchObservedRunningTime="2026-11-01 20:34:11.858209153 +0000 UTC m=+1859.532468408" Nov 01 20:34:12.625627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:12.625599 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:13.088288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:13.088247 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:14.086598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.086561 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m"] Nov 01 20:34:14.091223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.091202 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.147052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.147011 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m"] Nov 01 20:34:14.157815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.157768 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.158016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.157828 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.158080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.158045 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcc9p\" (UniqueName: \"kubernetes.io/projected/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-kube-api-access-bcc9p\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.190233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.190197 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:34:14.190233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.190199 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:34:14.190434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.190203 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:34:14.192348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.192313 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:34:14.259657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.259621 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.259839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.259670 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.259839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.259774 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcc9p\" (UniqueName: \"kubernetes.io/projected/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-kube-api-access-bcc9p\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.260410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.260382 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.262157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.262139 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.315961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.315929 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcc9p\" (UniqueName: \"kubernetes.io/projected/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-kube-api-access-bcc9p\") pod \"maas-api-revoke-sub-90ddc240-sdb7m\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.405713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.405630 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:14.783453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:14.783421 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m"] Nov 01 20:34:14.783639 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:34:14.783603 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9daa6ec7_64fa_4afc_b307_2f7e627a6ce8.slice/crio-6b3d8974bfc28a1c969ed28f57bc9cc5f9639f3a4c750bc6b288b85ab0e3b89e WatchSource:0}: Error finding container 6b3d8974bfc28a1c969ed28f57bc9cc5f9639f3a4c750bc6b288b85ab0e3b89e: Status 404 returned error can't find the container with id 6b3d8974bfc28a1c969ed28f57bc9cc5f9639f3a4c750bc6b288b85ab0e3b89e Nov 01 20:34:15.777441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:15.777406 5633 generic.go:356] "Generic (PLEG): container finished" podID="9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" containerID="09e896110dd1749f62670e3d9fdc44898ae7f0b8315b7d5593687c4e1b75c06b" exitCode=0 Nov 01 20:34:15.777848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:15.777491 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" event={"ID":"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8","Type":"ContainerDied","Data":"09e896110dd1749f62670e3d9fdc44898ae7f0b8315b7d5593687c4e1b75c06b"} Nov 01 20:34:15.777848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:15.777550 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" event={"ID":"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8","Type":"ContainerStarted","Data":"6b3d8974bfc28a1c969ed28f57bc9cc5f9639f3a4c750bc6b288b85ab0e3b89e"} Nov 01 20:34:16.156455 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:34:16.156312 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:34:16.805747 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:16.805714 5633 scope.go:122] "RemoveContainer" containerID="66a066e32d2a8e36222ed22b4e3b7f7f447caa89a235cd1458b98a92f022b4bc" Nov 01 20:34:16.820919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:16.820892 5633 scope.go:122] "RemoveContainer" containerID="6abb650fb707fc87d761ee7286d8151f128bd5156dd9a576ab1f02f9d3f89a64" Nov 01 20:34:16.832662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:16.832616 5633 scope.go:122] "RemoveContainer" containerID="57039fa4d324b805ffe8aab43c5c7b2eaa6c2e0d3935baf9b650459a3d5f37a8" Nov 01 20:34:16.976602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:16.976491 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:17.048927 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.048829 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:34:17.117355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.104314 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-xks-ca\") pod \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " Nov 01 20:34:17.117355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.104963 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-kube-api-access-bcc9p\" (UniqueName: \"kubernetes.io/projected/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-kube-api-access-bcc9p\") pod \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " Nov 01 20:34:17.117355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.105401 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-openshift-ca\") pod \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\" (UID: \"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8\") " Nov 01 20:34:17.117355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.107781 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-openshift-ca" pod "9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" (UID: "9daa6ec7-64fa-4afc-b307-2f7e627a6ce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:34:17.202617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.191660 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-kube-api-access-bcc9p" pod "9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" (UID: "9daa6ec7-64fa-4afc-b307-2f7e627a6ce8"). InnerVolumeSpecName "kube-api-access-bcc9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:17.202617 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.191855 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-xks-ca" pod "9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" (UID: "9daa6ec7-64fa-4afc-b307-2f7e627a6ce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:34:17.209350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.207572 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcc9p\" (UniqueName: \"kubernetes.io/projected/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-kube-api-access-bcc9p\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:17.209350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.207610 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:17.209350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.207627 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:17.803642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.803600 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" event={"ID":"9daa6ec7-64fa-4afc-b307-2f7e627a6ce8","Type":"ContainerDied","Data":"6b3d8974bfc28a1c969ed28f57bc9cc5f9639f3a4c750bc6b288b85ab0e3b89e"} Nov 01 20:34:17.803989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.803966 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3d8974bfc28a1c969ed28f57bc9cc5f9639f3a4c750bc6b288b85ab0e3b89e" Nov 01 20:34:17.804173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.804043 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m" Nov 01 20:34:17.922518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:17.922480 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:20.625512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:20.625467 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:20.639569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:20.639534 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:20.846583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:20.846553 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:22.850416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:22.850379 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:22.991661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:22.991614 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:24.351013 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:24.350970 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:24.812637 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:24.812601 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:25.826395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:25.826319 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:26.009680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:26.009643 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:26.132535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:26.132438 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:27.766799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:27.766760 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:28.031051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:28.030956 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:28.229313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:28.229276 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:29.573311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:29.573278 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:30.741193 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.741161 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58777c5b7f-wl887"] Nov 01 20:34:30.742111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.742084 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" containerName="revoke-keys" Nov 01 20:34:30.742291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.742277 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" containerName="revoke-keys" Nov 01 20:34:30.742661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.742636 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" containerName="revoke-keys" Nov 01 20:34:30.748084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.748063 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.803377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.803322 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58777c5b7f-wl887"] Nov 01 20:34:30.830551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.830515 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-routing-overlay-volume\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.830818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.830670 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9h5\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-kube-api-access-jn9h5\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.830899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.830820 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-plugins-config-volume\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.830899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.830869 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.846526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.846487 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 20:34:30.846526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.846510 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 20:34:30.846526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.846498 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ca90c301433\"" Nov 01 20:34:30.846814 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.846564 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 20:34:30.850005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.849984 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8ed0bcb9390\"" Nov 01 20:34:30.850005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.849985 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qz59c\"" Nov 01 20:34:30.850005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.850000 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 20:34:30.932199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.932162 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-routing-overlay-volume\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.932438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.932248 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9h5\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-kube-api-access-jn9h5\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.932438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.932352 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-plugins-config-volume\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.932438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.932386 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.933015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.932981 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-routing-overlay-volume\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.933156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.933083 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-plugins-config-volume\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.935127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.935104 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:30.990502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:30.990472 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9h5\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-kube-api-access-jn9h5\") pod \"payload-processing-external-model-58777c5b7f-wl887\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:31.006264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:31.006168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:31.061910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:31.061873 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:31.438729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:31.438701 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58777c5b7f-wl887"] Nov 01 20:34:31.439255 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:34:31.439225 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37eff729_76d4_41a0_a88c_5cf03cce0537.slice/crio-7ba2faaebee4367e27f66ffcb0c135726459aa84418ade4de5baa6778e70f13a WatchSource:0}: Error finding container 7ba2faaebee4367e27f66ffcb0c135726459aa84418ade4de5baa6778e70f13a: Status 404 returned error can't find the container with id 7ba2faaebee4367e27f66ffcb0c135726459aa84418ade4de5baa6778e70f13a Nov 01 20:34:31.943082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:31.943039 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" event={"ID":"37eff729-76d4-41a0-a88c-5cf03cce0537","Type":"ContainerStarted","Data":"08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab"} Nov 01 20:34:31.943571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:31.943105 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" event={"ID":"37eff729-76d4-41a0-a88c-5cf03cce0537","Type":"ContainerStarted","Data":"7ba2faaebee4367e27f66ffcb0c135726459aa84418ade4de5baa6778e70f13a"} Nov 01 20:34:31.943571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:31.943132 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:32.062049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.061991 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" podStartSLOduration=2.061974312 podStartE2EDuration="2.061974312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:32.058133662 +0000 UTC m=+1879.732392932" watchObservedRunningTime="2026-11-01 20:34:32.061974312 +0000 UTC m=+1879.736233591" Nov 01 20:34:32.112272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.112241 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc"] Nov 01 20:34:32.116848 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.116814 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.119610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.119583 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4"] Nov 01 20:34:32.123815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.123793 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.149861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.149827 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.149880 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.149995 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631d8092-14c4-49b7-b045-94f8077fce9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.150024 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.150055 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.150117 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.150227 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631d8092-14c4-49b7-b045-94f8077fce9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.150291 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.150392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.150379 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmdw\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-kube-api-access-qbmdw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.171102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.170962 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\"" Nov 01 20:34:32.179374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.179294 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc"] Nov 01 20:34:32.228991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.228947 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4"] Nov 01 20:34:32.251663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.251619 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ds4l\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-kube-api-access-2ds4l\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.251818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.251717 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.251818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.251761 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.251911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.251830 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.251911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.251871 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.251919 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631d8092-14c4-49b7-b045-94f8077fce9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.251974 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252016 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.252109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252055 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252083 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252140 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631d8092-14c4-49b7-b045-94f8077fce9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252228 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252284 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252369 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252369 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.252442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252430 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.252691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252508 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.252691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252573 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.252691 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252604 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.252864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252719 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.252864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.252758 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmdw\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-kube-api-access-qbmdw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.253035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.253000 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631d8092-14c4-49b7-b045-94f8077fce9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.253551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.253528 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.254801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.254769 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.255289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.255266 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631d8092-14c4-49b7-b045-94f8077fce9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.310129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.310091 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmdw\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-kube-api-access-qbmdw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.310268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.310210 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kjfvc\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.354218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354176 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354243 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354259 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354285 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354323 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354381 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354436 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ds4l\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-kube-api-access-2ds4l\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354492 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354532 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354753 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.354861 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.354785 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.355143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.355108 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.355311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.355285 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.355472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.355456 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.357032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.357013 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.357256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.357239 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.411321 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.411287 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.414701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.414671 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ds4l\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-kube-api-access-2ds4l\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfjk4\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.442765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.442733 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:32.446549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.446528 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:32.511675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.511546 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:32.681055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.681016 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:32.823643 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.823540 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:32.882062 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:34:32.882031 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631d8092_14c4_49b7_b045_94f8077fce9f.slice/crio-b0be211369fc7f7c467ed0aee4636babd04c9363f74d5c50807fe4ace4aef940 WatchSource:0}: Error finding container b0be211369fc7f7c467ed0aee4636babd04c9363f74d5c50807fe4ace4aef940: Status 404 returned error can't find the container with id b0be211369fc7f7c467ed0aee4636babd04c9363f74d5c50807fe4ace4aef940 Nov 01 20:34:32.882685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.882667 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4"] Nov 01 20:34:32.882945 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:34:32.882916 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ec2309_8a59_4003_a19b_9e54529e6ff6.slice/crio-cb34f109390a1200117f115f2b5012eb1396bc0ba6a6b2184f28e1bb94256aba WatchSource:0}: Error finding container cb34f109390a1200117f115f2b5012eb1396bc0ba6a6b2184f28e1bb94256aba: Status 404 returned error can't find the container with id cb34f109390a1200117f115f2b5012eb1396bc0ba6a6b2184f28e1bb94256aba Nov 01 20:34:32.884993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.884955 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:34:32.885061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.885044 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:34:32.885097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.885083 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:34:32.885680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.885652 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:34:32.885751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.885711 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:34:32.885751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.885740 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:34:32.887695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.887666 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc"] Nov 01 20:34:32.952268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.952227 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" event={"ID":"631d8092-14c4-49b7-b045-94f8077fce9f","Type":"ContainerStarted","Data":"b0be211369fc7f7c467ed0aee4636babd04c9363f74d5c50807fe4ace4aef940"} Nov 01 20:34:32.953753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:32.953724 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" event={"ID":"a1ec2309-8a59-4003-a19b-9e54529e6ff6","Type":"ContainerStarted","Data":"cb34f109390a1200117f115f2b5012eb1396bc0ba6a6b2184f28e1bb94256aba"} Nov 01 20:34:33.968990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:33.968379 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" event={"ID":"631d8092-14c4-49b7-b045-94f8077fce9f","Type":"ContainerStarted","Data":"1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2"} Nov 01 20:34:33.971188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:33.971124 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" event={"ID":"a1ec2309-8a59-4003-a19b-9e54529e6ff6","Type":"ContainerStarted","Data":"14d8ada7a9f3ae4ed8f778fd7b7a99d4dcc97f4a2e1bfbf1fed5d931673654ff"} Nov 01 20:34:34.135590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.135507 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podStartSLOduration=2.135483237 podStartE2EDuration="2.135483237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:34.13291662 +0000 UTC m=+1881.807175889" watchObservedRunningTime="2026-11-01 20:34:34.135483237 +0000 UTC m=+1881.809742493" Nov 01 20:34:34.249657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.249536 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podStartSLOduration=2.249515245 podStartE2EDuration="2.249515245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:34.247627281 +0000 UTC m=+1881.921886542" watchObservedRunningTime="2026-11-01 20:34:34.249515245 +0000 UTC m=+1881.923774502" Nov 01 20:34:34.443044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.443011 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:34.444655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.444632 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:34.444824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.444801 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:34.446710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.446693 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:34.447708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.447688 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 20:34:34.447767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:34.447728 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 20:34:35.443562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:35.443469 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:35.443562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:35.443540 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:35.447807 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:35.447781 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 20:34:35.447882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:35.447833 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 20:34:36.443961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:36.443918 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:36.444460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:36.443984 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:36.447250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:36.447227 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 20:34:36.447311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:36.447274 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 20:34:37.443911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:37.443870 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:37.444117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:37.443940 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:37.447240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:37.447217 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 20:34:37.447304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:37.447261 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 20:34:37.581577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:37.581530 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:38.443871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:38.443833 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:38.444032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:38.443886 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:38.447572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:38.447546 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 20:34:38.447928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:38.447589 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 20:34:39.443836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:39.443790 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:39.444050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:39.443858 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:39.447423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:39.447396 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 20:34:39.447559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:39.447458 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 20:34:40.443314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:40.443276 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:40.443892 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:40.443360 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:40.447702 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:40.447668 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 20:34:40.447849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:40.447718 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 20:34:41.444002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:41.443957 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 20:34:41.444466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:41.444049 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 20:34:42.488588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.488533 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:34:42.489117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.488621 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:34:42.494055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.493936 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:42.683107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.683066 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:42.684019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.683946 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:42.684975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.684952 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:34:42.713720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.713683 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:42.714043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.714018 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:42.715008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.714991 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:34:42.958083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:42.957989 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:34:44.617094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:44.617013 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:44.624474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:44.624442 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd"] Nov 01 20:34:45.072413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:45.072373 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:46.159778 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:34:46.159735 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:34:47.191266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:47.191229 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:50.134358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:50.134254 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:52.026875 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:52.026623 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:56.801777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:56.801738 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:34:59.646798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.646758 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" podUID="828e7588-20ab-4b25-b265-4cfa5b4c7ccd" containerName="main" containerID="cri-o://e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf" gracePeriod=45 Nov 01 20:34:59.915486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.915411 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:34:59.976990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.976955 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-home\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-home\") pod \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " Nov 01 20:34:59.977184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977047 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-model-cache\") pod \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " Nov 01 20:34:59.977184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977124 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tmp-dir\") pod \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " Nov 01 20:34:59.977184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977175 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-home" pod "828e7588-20ab-4b25-b265-4cfa5b4c7ccd" (UID: "828e7588-20ab-4b25-b265-4cfa5b4c7ccd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:59.977412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977214 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-kube-api-access-fjrsv\" (UniqueName: \"kubernetes.io/projected/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-kube-api-access-fjrsv\") pod \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " Nov 01 20:34:59.977412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977367 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-model-cache" pod "828e7588-20ab-4b25-b265-4cfa5b4c7ccd" (UID: "828e7588-20ab-4b25-b265-4cfa5b4c7ccd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:59.977412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977382 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tmp-dir" pod "828e7588-20ab-4b25-b265-4cfa5b4c7ccd" (UID: "828e7588-20ab-4b25-b265-4cfa5b4c7ccd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:59.977412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977387 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tls-certs\" (UniqueName: \"kubernetes.io/secret/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tls-certs\") pod \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " Nov 01 20:34:59.977641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977432 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-dshm\") pod \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\" (UID: \"828e7588-20ab-4b25-b265-4cfa5b4c7ccd\") " Nov 01 20:34:59.978043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.977975 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:59.978043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.978003 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:59.978043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.978019 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:59.980076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.980027 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-kube-api-access-fjrsv" pod "828e7588-20ab-4b25-b265-4cfa5b4c7ccd" (UID: "828e7588-20ab-4b25-b265-4cfa5b4c7ccd"). InnerVolumeSpecName "kube-api-access-fjrsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:59.980185 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.980161 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-dshm" pod "828e7588-20ab-4b25-b265-4cfa5b4c7ccd" (UID: "828e7588-20ab-4b25-b265-4cfa5b4c7ccd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:59.980456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:34:59.980408 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tls-certs" pod "828e7588-20ab-4b25-b265-4cfa5b4c7ccd" (UID: "828e7588-20ab-4b25-b265-4cfa5b4c7ccd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:35:00.079754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.079715 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjrsv\" (UniqueName: \"kubernetes.io/projected/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-kube-api-access-fjrsv\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:00.079754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.079748 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:00.079754 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.079763 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828e7588-20ab-4b25-b265-4cfa5b4c7ccd-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:00.271902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.271866 5633 generic.go:356] "Generic (PLEG): container finished" podID="828e7588-20ab-4b25-b265-4cfa5b4c7ccd" containerID="e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf" exitCode=0 Nov 01 20:35:00.272120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.271911 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" Nov 01 20:35:00.272120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.271951 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" event={"ID":"828e7588-20ab-4b25-b265-4cfa5b4c7ccd","Type":"ContainerDied","Data":"e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf"} Nov 01 20:35:00.272120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.272004 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" event={"ID":"828e7588-20ab-4b25-b265-4cfa5b4c7ccd","Type":"ContainerDied","Data":"b6d20958865c7cd66d213a08e2d1954e6205978f9344470de47508dd857fa631"} Nov 01 20:35:00.272120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.272030 5633 scope.go:122] "RemoveContainer" containerID="e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf" Nov 01 20:35:00.284146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.284125 5633 scope.go:122] "RemoveContainer" containerID="e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf" Nov 01 20:35:00.284538 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:35:00.284515 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf\": container with ID starting with e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf not found: ID does not exist" containerID="e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf" Nov 01 20:35:00.284620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.284552 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf"} err="failed to get container status \"e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf\": rpc error: code = NotFound desc = could not find container \"e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf\": container with ID starting with e367b7889b18b8be54646e77ed43b5be869b143c3a12eee20887583fda512acf not found: ID does not exist" Nov 01 20:35:00.445349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.438224 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd"] Nov 01 20:35:00.485897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.485856 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd"] Nov 01 20:35:00.835396 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:00.835317 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-f154f7/explicit-model-f154f7-kserve-5b89475549-6ffnd" podUID="828e7588-20ab-4b25-b265-4cfa5b4c7ccd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:35:01.450292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:01.450245 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:02.468619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:02.468585 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828e7588-20ab-4b25-b265-4cfa5b4c7ccd" path="/var/lib/kubelet/pods/828e7588-20ab-4b25-b265-4cfa5b4c7ccd/volumes" Nov 01 20:35:03.247904 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:03.247863 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:35:06.164554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:06.164518 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:10.891110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:10.891075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:12.051396 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:12.051361 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k"] Nov 01 20:35:12.100664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:12.100627 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-716d0712-n2t8k"] Nov 01 20:35:12.468696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:12.468652 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60c2919-e2b2-4b14-9af2-16c84bcdf826" path="/var/lib/kubelet/pods/a60c2919-e2b2-4b14-9af2-16c84bcdf826/volumes" Nov 01 20:35:15.590277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:15.590241 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:16.157325 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:35:16.157289 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:35:16.987605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:16.987575 5633 scope.go:122] "RemoveContainer" containerID="294bc6c36303f65ab0a687240c0590cdf85c0f2c7ba0ac892ae3dc10f377d934" Nov 01 20:35:17.484452 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.484417 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn"] Nov 01 20:35:17.485044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.485029 5633 cpu_manager.go:414] "containerMap: removing container" podUID="828e7588-20ab-4b25-b265-4cfa5b4c7ccd" containerName="main" Nov 01 20:35:17.485097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.485047 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828e7588-20ab-4b25-b265-4cfa5b4c7ccd" containerName="main" Nov 01 20:35:17.485220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.485211 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="828e7588-20ab-4b25-b265-4cfa5b4c7ccd" containerName="main" Nov 01 20:35:17.490519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.490495 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.553955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.553920 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2"] Nov 01 20:35:17.559128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.559107 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.601081 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601038 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601172 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601237 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601294 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601381 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601413 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601461 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601522 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szv7g\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-kube-api-access-szv7g\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.601693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.601572 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.630679 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.626994 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn"] Nov 01 20:35:17.632881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.632850 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2"] Nov 01 20:35:17.703194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703159 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.703394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703201 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.703394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703223 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.703487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703443 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.703550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703512 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.703607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703577 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.703666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703622 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.703666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703656 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.703782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703689 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.703782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703715 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.703782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703759 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.703946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703799 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4br\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-kube-api-access-jf4br\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.703946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703922 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.704048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703971 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.704048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.703980 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.704048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704040 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szv7g\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-kube-api-access-szv7g\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.704204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704049 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.704204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704063 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.704204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704127 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.704204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704160 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.704428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704208 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.704542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704510 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.704846 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.704721 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.706252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.706224 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.706442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.706424 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.760987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.760910 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.761171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.761114 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szv7g\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-kube-api-access-szv7g\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6xnbn\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.805235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805198 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805235 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805268 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4br\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-kube-api-access-jf4br\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805314 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805369 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805407 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805460 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805529 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805560 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805698 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805836 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805880 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:17.805962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805919 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.805962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.805935 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.806381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.806239 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.807850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.807829 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.808295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.808274 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.862802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.862115 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.862991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.862955 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4br\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-kube-api-access-jf4br\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lc5x2\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:17.874045 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:17.873795 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:18.237602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.237572 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn"] Nov 01 20:35:18.237959 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:35:18.237858 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f494f1_e9ae_427e_bae3_bf80db4fb48a.slice/crio-7691ec880b8b2efc17cb9e79d707c04cb087c5d492491f96a6a279cb8179519d WatchSource:0}: Error finding container 7691ec880b8b2efc17cb9e79d707c04cb087c5d492491f96a6a279cb8179519d: Status 404 returned error can't find the container with id 7691ec880b8b2efc17cb9e79d707c04cb087c5d492491f96a6a279cb8179519d Nov 01 20:35:18.240431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.240387 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:18.240544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.240466 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:18.240544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.240492 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:18.289212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.289185 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2"] Nov 01 20:35:18.297139 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:35:18.297112 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd04fc0_d37e_4de9_8f6f_aa319f847b4e.slice/crio-50ab5a5fd80b814406407e80b4d7247facde199c509cde12c6a647beceb4aee5 WatchSource:0}: Error finding container 50ab5a5fd80b814406407e80b4d7247facde199c509cde12c6a647beceb4aee5: Status 404 returned error can't find the container with id 50ab5a5fd80b814406407e80b4d7247facde199c509cde12c6a647beceb4aee5 Nov 01 20:35:18.299909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.299872 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:18.300019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.299947 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:18.300019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.299983 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:18.417000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.416965 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" event={"ID":"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e","Type":"ContainerStarted","Data":"2a3aac4129ef88601146fde75c09918f439021d710f469ee3944a2e6badd57dc"} Nov 01 20:35:18.417287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.417260 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" event={"ID":"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e","Type":"ContainerStarted","Data":"50ab5a5fd80b814406407e80b4d7247facde199c509cde12c6a647beceb4aee5"} Nov 01 20:35:18.420959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.419324 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" event={"ID":"f8f494f1-e9ae-427e-bae3-bf80db4fb48a","Type":"ContainerStarted","Data":"716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3"} Nov 01 20:35:18.420959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.419432 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" event={"ID":"f8f494f1-e9ae-427e-bae3-bf80db4fb48a","Type":"ContainerStarted","Data":"7691ec880b8b2efc17cb9e79d707c04cb087c5d492491f96a6a279cb8179519d"} Nov 01 20:35:18.539093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.538957 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podStartSLOduration=1.538942627 podStartE2EDuration="1.538942627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:18.53107237 +0000 UTC m=+1926.205331625" watchObservedRunningTime="2026-11-01 20:35:18.538942627 +0000 UTC m=+1926.213201884" Nov 01 20:35:18.650414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.650346 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podStartSLOduration=1.65030792 podStartE2EDuration="1.65030792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:18.646735982 +0000 UTC m=+1926.320995238" watchObservedRunningTime="2026-11-01 20:35:18.65030792 +0000 UTC m=+1926.324567177" Nov 01 20:35:18.807379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.807265 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:18.875170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:18.875119 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:19.814275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:19.814236 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:19.814944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:19.814900 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:19.879687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:19.879642 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:19.879984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:19.879936 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:20.476482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:20.476444 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:20.809620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:20.809519 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:20.809899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:20.809871 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:20.875414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:20.875384 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:20.875948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:20.875891 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:21.810354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:21.809745 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:21.810354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:21.809819 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:21.874729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:21.874688 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:21.874915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:21.874746 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:22.817358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:22.816788 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:22.817358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:22.816851 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:22.876873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:22.876837 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:22.877200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:22.877152 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:23.068611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:23.066897 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq"] Nov 01 20:35:23.144367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:23.142544 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f896a4-vbnwq"] Nov 01 20:35:23.815358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:23.814788 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:23.815358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:23.814866 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:23.875605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:23.875564 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:23.876015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:23.875618 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:24.312359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.297980 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz"] Nov 01 20:35:24.321906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.321874 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.368010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.367976 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz"] Nov 01 20:35:24.450358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.444803 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:35:24.450358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.444887 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:35:24.450358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.444955 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:35:24.450358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.445052 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:35:24.476792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.474721 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.476792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.474904 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqkj\" (UniqueName: \"kubernetes.io/projected/695ff339-3f54-45dd-a068-161276bc9b70-kube-api-access-gqqkj\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.476792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.475062 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.480961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.480928 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a30213-3dbd-47df-a4ef-0e7836f4c4ce" path="/var/lib/kubelet/pods/01a30213-3dbd-47df-a4ef-0e7836f4c4ce/volumes" Nov 01 20:35:24.576179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.576074 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.576457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.576207 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqkj\" (UniqueName: \"kubernetes.io/projected/695ff339-3f54-45dd-a068-161276bc9b70-kube-api-access-gqqkj\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.576457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.576267 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.576956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.576933 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.578840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.578816 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.633511 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.633474 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqkj\" (UniqueName: \"kubernetes.io/projected/695ff339-3f54-45dd-a068-161276bc9b70-kube-api-access-gqqkj\") pod \"maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.670534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.670495 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:24.839557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.839474 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:24.839824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.839791 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:24.875142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.875100 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:24.875373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:24.875161 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:25.123137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.123026 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz"] Nov 01 20:35:25.130173 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:35:25.130126 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695ff339_3f54_45dd_a068_161276bc9b70.slice/crio-f661fdc5349713199caf3b31e4234a2f8bb0b86674914b0fbfb52aa52dc7f4db WatchSource:0}: Error finding container f661fdc5349713199caf3b31e4234a2f8bb0b86674914b0fbfb52aa52dc7f4db: Status 404 returned error can't find the container with id f661fdc5349713199caf3b31e4234a2f8bb0b86674914b0fbfb52aa52dc7f4db Nov 01 20:35:25.573943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.573900 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" event={"ID":"695ff339-3f54-45dd-a068-161276bc9b70","Type":"ContainerStarted","Data":"eb55f1f8426131881d1310538e1cabdbdeed87729a41570b0bc734183ab7ad53"} Nov 01 20:35:25.574156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.573964 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" event={"ID":"695ff339-3f54-45dd-a068-161276bc9b70","Type":"ContainerStarted","Data":"f661fdc5349713199caf3b31e4234a2f8bb0b86674914b0fbfb52aa52dc7f4db"} Nov 01 20:35:25.690650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.690579 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" podStartSLOduration=1.690558742 podStartE2EDuration="1.690558742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:25.685213507 +0000 UTC m=+1933.359472779" watchObservedRunningTime="2026-11-01 20:35:25.690558742 +0000 UTC m=+1933.364817997" Nov 01 20:35:25.806805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.806718 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:25.806805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.806788 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:25.875578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.875541 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:25.875752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:25.875611 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:26.033758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:26.033710 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:26.587879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:26.587842 5633 generic.go:356] "Generic (PLEG): container finished" podID="695ff339-3f54-45dd-a068-161276bc9b70" containerID="eb55f1f8426131881d1310538e1cabdbdeed87729a41570b0bc734183ab7ad53" exitCode=0 Nov 01 20:35:26.588436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:26.587907 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" event={"ID":"695ff339-3f54-45dd-a068-161276bc9b70","Type":"ContainerDied","Data":"eb55f1f8426131881d1310538e1cabdbdeed87729a41570b0bc734183ab7ad53"} Nov 01 20:35:26.806961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:26.806911 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 01 20:35:26.807171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:26.806973 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 01 20:35:26.874801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:26.874699 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 01 20:35:26.874801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:26.874774 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 01 20:35:27.089283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:27.089242 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:27.539941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:27.539209 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t"] Nov 01 20:35:27.656414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:27.655277 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl"] Nov 01 20:35:27.939687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:27.918784 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:28.052234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.051834 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:28.065103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.065043 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:28.142680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.140941 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-openshift-ca\") pod \"695ff339-3f54-45dd-a068-161276bc9b70\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " Nov 01 20:35:28.142680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.141037 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695ff339-3f54-45dd-a068-161276bc9b70-kube-api-access-gqqkj\" (UniqueName: \"kubernetes.io/projected/695ff339-3f54-45dd-a068-161276bc9b70-kube-api-access-gqqkj\") pod \"695ff339-3f54-45dd-a068-161276bc9b70\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " Nov 01 20:35:28.142680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.141104 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-xks-ca\") pod \"695ff339-3f54-45dd-a068-161276bc9b70\" (UID: \"695ff339-3f54-45dd-a068-161276bc9b70\") " Nov 01 20:35:28.142680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.142126 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-openshift-ca" pod "695ff339-3f54-45dd-a068-161276bc9b70" (UID: "695ff339-3f54-45dd-a068-161276bc9b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:35:28.148728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.148676 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-xks-ca" pod "695ff339-3f54-45dd-a068-161276bc9b70" (UID: "695ff339-3f54-45dd-a068-161276bc9b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:35:28.148728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.148704 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695ff339-3f54-45dd-a068-161276bc9b70-kube-api-access-gqqkj" pod "695ff339-3f54-45dd-a068-161276bc9b70" (UID: "695ff339-3f54-45dd-a068-161276bc9b70"). InnerVolumeSpecName "kube-api-access-gqqkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:28.243075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.242963 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:28.243075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.243009 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqkj\" (UniqueName: \"kubernetes.io/projected/695ff339-3f54-45dd-a068-161276bc9b70-kube-api-access-gqqkj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:28.243075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.243027 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695ff339-3f54-45dd-a068-161276bc9b70-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:28.446605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.446481 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:28.649030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.648988 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" event={"ID":"695ff339-3f54-45dd-a068-161276bc9b70","Type":"ContainerDied","Data":"f661fdc5349713199caf3b31e4234a2f8bb0b86674914b0fbfb52aa52dc7f4db"} Nov 01 20:35:28.649492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.649470 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f661fdc5349713199caf3b31e4234a2f8bb0b86674914b0fbfb52aa52dc7f4db" Nov 01 20:35:28.649673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.649380 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz" Nov 01 20:35:28.805857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.805810 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:28.809719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.807542 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:28.809719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.809544 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:35:28.878436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.878363 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:35:28.878620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.878440 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:35:28.990356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.989853 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:28.994353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.991143 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:29.002359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:28.997785 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:35:30.666213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:30.666181 5633 generic.go:356] "Generic (PLEG): container finished" podID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerID="bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1" exitCode=0 Nov 01 20:35:30.666690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:30.666248 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" event={"ID":"0e5b127f-1db8-4a60-91c1-7bf056c693a3","Type":"ContainerDied","Data":"bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1"} Nov 01 20:35:30.667121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:30.667101 5633 scope.go:122] "RemoveContainer" containerID="bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1" Nov 01 20:35:30.668574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:30.668548 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:30.668644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:30.668617 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:30.668701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:30.668649 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:35:31.229963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:31.229928 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:31.675453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:31.675364 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" event={"ID":"0e5b127f-1db8-4a60-91c1-7bf056c693a3","Type":"ContainerStarted","Data":"eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3"} Nov 01 20:35:32.063043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.063008 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:35:32.064568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.064541 5633 patch_prober.go:29] interesting pod/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 20:35:32.064696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.064601 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 20:35:32.540426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.540377 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" podUID="73ccbde2-d089-47b0-8582-e8df179b9b5f" containerName="payload-processing" containerID="cri-o://a298d807f326f2facf2f7d0ad945f33ebb63edae3a85ddb6392404cb28e310dd" gracePeriod=25 Nov 01 20:35:32.656307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.656268 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" podUID="b28207c7-f498-43ed-b16f-5eeea8fd28d5" containerName="payload-pre-processing" containerID="cri-o://0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6" gracePeriod=25 Nov 01 20:35:32.708468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.705687 5633 generic.go:356] "Generic (PLEG): container finished" podID="73ccbde2-d089-47b0-8582-e8df179b9b5f" containerID="a298d807f326f2facf2f7d0ad945f33ebb63edae3a85ddb6392404cb28e310dd" exitCode=0 Nov 01 20:35:32.708468 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.706758 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" event={"ID":"73ccbde2-d089-47b0-8582-e8df179b9b5f","Type":"ContainerDied","Data":"a298d807f326f2facf2f7d0ad945f33ebb63edae3a85ddb6392404cb28e310dd"} Nov 01 20:35:32.772577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.772413 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:32.869522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.869481 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:35:32.929520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.929480 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ccbde2-d089-47b0-8582-e8df179b9b5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ccbde2-d089-47b0-8582-e8df179b9b5f-plugins-config-volume\") pod \"73ccbde2-d089-47b0-8582-e8df179b9b5f\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " Nov 01 20:35:32.929711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.929669 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ccbde2-d089-47b0-8582-e8df179b9b5f-kube-api-access-2bpmp\" (UniqueName: \"kubernetes.io/projected/73ccbde2-d089-47b0-8582-e8df179b9b5f-kube-api-access-2bpmp\") pod \"73ccbde2-d089-47b0-8582-e8df179b9b5f\" (UID: \"73ccbde2-d089-47b0-8582-e8df179b9b5f\") " Nov 01 20:35:32.929955 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.929908 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ccbde2-d089-47b0-8582-e8df179b9b5f-plugins-config-volume" pod "73ccbde2-d089-47b0-8582-e8df179b9b5f" (UID: "73ccbde2-d089-47b0-8582-e8df179b9b5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:35:32.930424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.930402 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ccbde2-d089-47b0-8582-e8df179b9b5f-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:32.933532 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:32.933505 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ccbde2-d089-47b0-8582-e8df179b9b5f-kube-api-access-2bpmp" pod "73ccbde2-d089-47b0-8582-e8df179b9b5f" (UID: "73ccbde2-d089-47b0-8582-e8df179b9b5f"). InnerVolumeSpecName "kube-api-access-2bpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:33.033229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.033194 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bpmp\" (UniqueName: \"kubernetes.io/projected/73ccbde2-d089-47b0-8582-e8df179b9b5f-kube-api-access-2bpmp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:33.039026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.038994 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:35:33.063142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.063108 5633 patch_prober.go:29] interesting pod/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 20:35:33.063311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.063178 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 20:35:33.134494 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.134412 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b28207c7-f498-43ed-b16f-5eeea8fd28d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28207c7-f498-43ed-b16f-5eeea8fd28d5-plugins-config-volume\") pod \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " Nov 01 20:35:33.134494 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.134488 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28207c7-f498-43ed-b16f-5eeea8fd28d5-kube-api-access-45kwj\" (UniqueName: \"kubernetes.io/projected/b28207c7-f498-43ed-b16f-5eeea8fd28d5-kube-api-access-45kwj\") pod \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\" (UID: \"b28207c7-f498-43ed-b16f-5eeea8fd28d5\") " Nov 01 20:35:33.134811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.134789 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28207c7-f498-43ed-b16f-5eeea8fd28d5-plugins-config-volume" pod "b28207c7-f498-43ed-b16f-5eeea8fd28d5" (UID: "b28207c7-f498-43ed-b16f-5eeea8fd28d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:35:33.135083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.135069 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28207c7-f498-43ed-b16f-5eeea8fd28d5-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:33.136850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.136830 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28207c7-f498-43ed-b16f-5eeea8fd28d5-kube-api-access-45kwj" pod "b28207c7-f498-43ed-b16f-5eeea8fd28d5" (UID: "b28207c7-f498-43ed-b16f-5eeea8fd28d5"). InnerVolumeSpecName "kube-api-access-45kwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:33.236959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.236921 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45kwj\" (UniqueName: \"kubernetes.io/projected/b28207c7-f498-43ed-b16f-5eeea8fd28d5-kube-api-access-45kwj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:33.720305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.720278 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" Nov 01 20:35:33.720851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.720318 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t" event={"ID":"73ccbde2-d089-47b0-8582-e8df179b9b5f","Type":"ContainerDied","Data":"67368e1a4f07ea72b72d75213c2ec4a29894f2c1ae0448a12850958e8eb075a7"} Nov 01 20:35:33.720851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.720427 5633 scope.go:122] "RemoveContainer" containerID="a298d807f326f2facf2f7d0ad945f33ebb63edae3a85ddb6392404cb28e310dd" Nov 01 20:35:33.723455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.723424 5633 generic.go:356] "Generic (PLEG): container finished" podID="b28207c7-f498-43ed-b16f-5eeea8fd28d5" containerID="0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6" exitCode=0 Nov 01 20:35:33.723604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.723508 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" Nov 01 20:35:33.723604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.723526 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" event={"ID":"b28207c7-f498-43ed-b16f-5eeea8fd28d5","Type":"ContainerDied","Data":"0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6"} Nov 01 20:35:33.723604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.723572 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl" event={"ID":"b28207c7-f498-43ed-b16f-5eeea8fd28d5","Type":"ContainerDied","Data":"5eefebddaccc4d56ca4b6f4480dc6c95b5f999a7438798ca9a947d49c57241c9"} Nov 01 20:35:33.736085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.736065 5633 scope.go:122] "RemoveContainer" containerID="0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6" Nov 01 20:35:33.746757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.746734 5633 scope.go:122] "RemoveContainer" containerID="0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6" Nov 01 20:35:33.747118 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:35:33.747096 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6\": container with ID starting with 0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6 not found: ID does not exist" containerID="0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6" Nov 01 20:35:33.747182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.747127 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6"} err="failed to get container status \"0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6\": rpc error: code = NotFound desc = could not find container \"0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6\": container with ID starting with 0fce0b37114bbec76f167974401c58522782be4cf3b56694c3eacb532303a3d6 not found: ID does not exist" Nov 01 20:35:33.905417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.886153 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t"] Nov 01 20:35:33.935277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:33.935231 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f154f7-58b8984679-7bv6t"] Nov 01 20:35:34.013247 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.013210 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl"] Nov 01 20:35:34.017449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.017423 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f154f7-5d856fbddc-ll4gl"] Nov 01 20:35:34.047619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.047586 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb"] Nov 01 20:35:34.051498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.051472 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22c3354-t9mkb"] Nov 01 20:35:34.063074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.063033 5633 patch_prober.go:29] interesting pod/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 20:35:34.063239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.063098 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 20:35:34.467749 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.467714 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ccbde2-d089-47b0-8582-e8df179b9b5f" path="/var/lib/kubelet/pods/73ccbde2-d089-47b0-8582-e8df179b9b5f/volumes" Nov 01 20:35:34.468156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.468144 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b28207c7-f498-43ed-b16f-5eeea8fd28d5" path="/var/lib/kubelet/pods/b28207c7-f498-43ed-b16f-5eeea8fd28d5/volumes" Nov 01 20:35:34.468593 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:34.468579 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65c914c-b5f9-40ee-af6b-66af24b3fff3" path="/var/lib/kubelet/pods/b65c914c-b5f9-40ee-af6b-66af24b3fff3/volumes" Nov 01 20:35:35.063007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:35.062972 5633 patch_prober.go:29] interesting pod/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 20:35:35.063473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:35.063028 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 20:35:35.812924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:35.812883 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2"] Nov 01 20:35:35.919384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:35.919307 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv"] Nov 01 20:35:36.019645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:36.019605 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:36.062546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:36.062509 5633 patch_prober.go:29] interesting pod/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 20:35:36.062706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:36.062563 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 20:35:36.212592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:36.212554 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:36.253418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:36.251474 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:35:36.334736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:36.334701 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:37.066883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:37.066851 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:35:37.067952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:37.067930 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:35:40.689243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:40.689205 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:40.813209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:40.813169 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" podUID="2d6ff52c-323d-47eb-af80-0160daf45272" containerName="payload-processing" containerID="cri-o://c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606" gracePeriod=25 Nov 01 20:35:40.920538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:40.920491 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" podUID="235939a2-8244-4b64-a584-2fc16f2ee829" containerName="payload-pre-processing" containerID="cri-o://a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97" gracePeriod=25 Nov 01 20:35:41.109105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.109076 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:35:41.219280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.219256 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:35:41.250294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.250182 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6ff52c-323d-47eb-af80-0160daf45272-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6ff52c-323d-47eb-af80-0160daf45272-plugins-config-volume\") pod \"2d6ff52c-323d-47eb-af80-0160daf45272\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " Nov 01 20:35:41.250515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.250383 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6ff52c-323d-47eb-af80-0160daf45272-kube-api-access-c7x79\" (UniqueName: \"kubernetes.io/projected/2d6ff52c-323d-47eb-af80-0160daf45272-kube-api-access-c7x79\") pod \"2d6ff52c-323d-47eb-af80-0160daf45272\" (UID: \"2d6ff52c-323d-47eb-af80-0160daf45272\") " Nov 01 20:35:41.250595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.250574 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6ff52c-323d-47eb-af80-0160daf45272-plugins-config-volume" pod "2d6ff52c-323d-47eb-af80-0160daf45272" (UID: "2d6ff52c-323d-47eb-af80-0160daf45272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:35:41.251066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.251044 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d6ff52c-323d-47eb-af80-0160daf45272-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:41.253049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.253022 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6ff52c-323d-47eb-af80-0160daf45272-kube-api-access-c7x79" pod "2d6ff52c-323d-47eb-af80-0160daf45272" (UID: "2d6ff52c-323d-47eb-af80-0160daf45272"). InnerVolumeSpecName "kube-api-access-c7x79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:41.351809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.351775 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235939a2-8244-4b64-a584-2fc16f2ee829-kube-api-access-8tm9p\" (UniqueName: \"kubernetes.io/projected/235939a2-8244-4b64-a584-2fc16f2ee829-kube-api-access-8tm9p\") pod \"235939a2-8244-4b64-a584-2fc16f2ee829\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " Nov 01 20:35:41.352020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.351913 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235939a2-8244-4b64-a584-2fc16f2ee829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235939a2-8244-4b64-a584-2fc16f2ee829-plugins-config-volume\") pod \"235939a2-8244-4b64-a584-2fc16f2ee829\" (UID: \"235939a2-8244-4b64-a584-2fc16f2ee829\") " Nov 01 20:35:41.352313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.352283 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235939a2-8244-4b64-a584-2fc16f2ee829-plugins-config-volume" pod "235939a2-8244-4b64-a584-2fc16f2ee829" (UID: "235939a2-8244-4b64-a584-2fc16f2ee829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:35:41.352428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.352411 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7x79\" (UniqueName: \"kubernetes.io/projected/2d6ff52c-323d-47eb-af80-0160daf45272-kube-api-access-c7x79\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:41.354263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.354236 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235939a2-8244-4b64-a584-2fc16f2ee829-kube-api-access-8tm9p" pod "235939a2-8244-4b64-a584-2fc16f2ee829" (UID: "235939a2-8244-4b64-a584-2fc16f2ee829"). InnerVolumeSpecName "kube-api-access-8tm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:41.453865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.453822 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tm9p\" (UniqueName: \"kubernetes.io/projected/235939a2-8244-4b64-a584-2fc16f2ee829-kube-api-access-8tm9p\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:41.453865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.453858 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235939a2-8244-4b64-a584-2fc16f2ee829-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:41.792771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.792731 5633 generic.go:356] "Generic (PLEG): container finished" podID="2d6ff52c-323d-47eb-af80-0160daf45272" containerID="c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606" exitCode=0 Nov 01 20:35:41.793300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.792786 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" Nov 01 20:35:41.793300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.792826 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" event={"ID":"2d6ff52c-323d-47eb-af80-0160daf45272","Type":"ContainerDied","Data":"c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606"} Nov 01 20:35:41.793300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.792895 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2" event={"ID":"2d6ff52c-323d-47eb-af80-0160daf45272","Type":"ContainerDied","Data":"0922b7d3c92dbeb845c5792d99f1fb31f797d6f7e1a42418a1ca70b65258049b"} Nov 01 20:35:41.793300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.792932 5633 scope.go:122] "RemoveContainer" containerID="c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606" Nov 01 20:35:41.794651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.794625 5633 generic.go:356] "Generic (PLEG): container finished" podID="235939a2-8244-4b64-a584-2fc16f2ee829" containerID="a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97" exitCode=0 Nov 01 20:35:41.794790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.794675 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" Nov 01 20:35:41.794893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.794807 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" event={"ID":"235939a2-8244-4b64-a584-2fc16f2ee829","Type":"ContainerDied","Data":"a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97"} Nov 01 20:35:41.794893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.794864 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv" event={"ID":"235939a2-8244-4b64-a584-2fc16f2ee829","Type":"ContainerDied","Data":"b06d4b5aec7bfb19cb1bbec1e3af448b141da6ca5e62929ff2e41680fd880a0e"} Nov 01 20:35:41.805558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.805538 5633 scope.go:122] "RemoveContainer" containerID="c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606" Nov 01 20:35:41.805830 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:35:41.805815 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606\": container with ID starting with c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606 not found: ID does not exist" containerID="c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606" Nov 01 20:35:41.805881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.805837 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606"} err="failed to get container status \"c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606\": rpc error: code = NotFound desc = could not find container \"c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606\": container with ID starting with c8ba15bc68d1c17b067d84bbb81fc4eae960b78ce0eaf4263c937fb787861606 not found: ID does not exist" Nov 01 20:35:41.805881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.805855 5633 scope.go:122] "RemoveContainer" containerID="a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97" Nov 01 20:35:41.815504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.815355 5633 scope.go:122] "RemoveContainer" containerID="a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97" Nov 01 20:35:41.815675 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:35:41.815653 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97\": container with ID starting with a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97 not found: ID does not exist" containerID="a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97" Nov 01 20:35:41.815743 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.815689 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97"} err="failed to get container status \"a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97\": rpc error: code = NotFound desc = could not find container \"a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97\": container with ID starting with a49b04d3dfcee067f01ea6a27ca3e083dc05cab0358bd56704142393ff9f4b97 not found: ID does not exist" Nov 01 20:35:41.957184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:41.957146 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv"] Nov 01 20:35:42.008564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:42.008528 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37811a-c7b4fd9f9-4cpxv"] Nov 01 20:35:42.115047 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:42.114958 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2"] Nov 01 20:35:42.118944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:42.118909 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f37811a-55bc874b65-mf4b2"] Nov 01 20:35:42.467832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:42.467799 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235939a2-8244-4b64-a584-2fc16f2ee829" path="/var/lib/kubelet/pods/235939a2-8244-4b64-a584-2fc16f2ee829/volumes" Nov 01 20:35:42.468190 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:42.468176 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6ff52c-323d-47eb-af80-0160daf45272" path="/var/lib/kubelet/pods/2d6ff52c-323d-47eb-af80-0160daf45272/volumes" Nov 01 20:35:45.383623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:45.383590 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:46.046215 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:46.046181 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw"] Nov 01 20:35:46.095851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:46.095816 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bc06ec-smlsw"] Nov 01 20:35:46.156267 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:35:46.156227 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:35:46.469152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:46.469120 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986f2f35-5197-40d4-943e-ee2bf738ee43" path="/var/lib/kubelet/pods/986f2f35-5197-40d4-943e-ee2bf738ee43/volumes" Nov 01 20:35:50.086324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:50.086283 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:54.884189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:54.884144 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:55.715620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.715579 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l"] Nov 01 20:35:55.716198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716184 5633 cpu_manager.go:414] "containerMap: removing container" podUID="695ff339-3f54-45dd-a068-161276bc9b70" containerName="revoke-keys" Nov 01 20:35:55.716246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716199 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695ff339-3f54-45dd-a068-161276bc9b70" containerName="revoke-keys" Nov 01 20:35:55.716246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716218 5633 cpu_manager.go:414] "containerMap: removing container" podUID="2d6ff52c-323d-47eb-af80-0160daf45272" containerName="payload-processing" Nov 01 20:35:55.716246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716224 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6ff52c-323d-47eb-af80-0160daf45272" containerName="payload-processing" Nov 01 20:35:55.716246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716242 5633 cpu_manager.go:414] "containerMap: removing container" podUID="73ccbde2-d089-47b0-8582-e8df179b9b5f" containerName="payload-processing" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716248 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ccbde2-d089-47b0-8582-e8df179b9b5f" containerName="payload-processing" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716256 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b28207c7-f498-43ed-b16f-5eeea8fd28d5" containerName="payload-pre-processing" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716260 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b28207c7-f498-43ed-b16f-5eeea8fd28d5" containerName="payload-pre-processing" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716284 5633 cpu_manager.go:414] "containerMap: removing container" podUID="235939a2-8244-4b64-a584-2fc16f2ee829" containerName="payload-pre-processing" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716289 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235939a2-8244-4b64-a584-2fc16f2ee829" containerName="payload-pre-processing" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716391 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="235939a2-8244-4b64-a584-2fc16f2ee829" containerName="payload-pre-processing" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716409 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="695ff339-3f54-45dd-a068-161276bc9b70" containerName="revoke-keys" Nov 01 20:35:55.716421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716420 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6ff52c-323d-47eb-af80-0160daf45272" containerName="payload-processing" Nov 01 20:35:55.716675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716427 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ccbde2-d089-47b0-8582-e8df179b9b5f" containerName="payload-processing" Nov 01 20:35:55.716675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.716432 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b28207c7-f498-43ed-b16f-5eeea8fd28d5" containerName="payload-pre-processing" Nov 01 20:35:55.721374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.721324 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.734185 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.734151 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.734314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.734255 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.734546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.734528 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsxf\" (UniqueName: \"kubernetes.io/projected/d17871a7-c69c-4a6c-a567-dbe08d346f20-kube-api-access-6gsxf\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.778816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.778780 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l"] Nov 01 20:35:55.822809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.822768 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:35:55.822809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.822777 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:35:55.822809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.822799 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:35:55.823055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.822828 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:35:55.836006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.835972 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsxf\" (UniqueName: \"kubernetes.io/projected/d17871a7-c69c-4a6c-a567-dbe08d346f20-kube-api-access-6gsxf\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.836194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.836039 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.836194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.836073 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.836728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.836706 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.838700 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.838679 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:55.894301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:55.894264 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsxf\" (UniqueName: \"kubernetes.io/projected/d17871a7-c69c-4a6c-a567-dbe08d346f20-kube-api-access-6gsxf\") pod \"maas-api-revoke-sub-734772f6-7hz4l\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:56.036614 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:56.036518 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:56.421138 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:35:56.421103 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17871a7_c69c_4a6c_a567_dbe08d346f20.slice/crio-a58036c27cb1a97acc84e192586c6577d9e8bf5daf6a424ec5368bcf52881717 WatchSource:0}: Error finding container a58036c27cb1a97acc84e192586c6577d9e8bf5daf6a424ec5368bcf52881717: Status 404 returned error can't find the container with id a58036c27cb1a97acc84e192586c6577d9e8bf5daf6a424ec5368bcf52881717 Nov 01 20:35:56.422628 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:56.422598 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l"] Nov 01 20:35:56.908652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:56.908613 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" event={"ID":"d17871a7-c69c-4a6c-a567-dbe08d346f20","Type":"ContainerStarted","Data":"6111bdc2da2278a05ac6674aaac4043f6990694909ccd86a9060132661bab7ad"} Nov 01 20:35:56.909075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:56.908680 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" event={"ID":"d17871a7-c69c-4a6c-a567-dbe08d346f20","Type":"ContainerStarted","Data":"a58036c27cb1a97acc84e192586c6577d9e8bf5daf6a424ec5368bcf52881717"} Nov 01 20:35:57.031389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:57.031314 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" podStartSLOduration=2.031298217 podStartE2EDuration="2.031298217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:57.02819357 +0000 UTC m=+1964.702452826" watchObservedRunningTime="2026-11-01 20:35:57.031298217 +0000 UTC m=+1964.705557473" Nov 01 20:35:57.917400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:57.917358 5633 generic.go:356] "Generic (PLEG): container finished" podID="d17871a7-c69c-4a6c-a567-dbe08d346f20" containerID="6111bdc2da2278a05ac6674aaac4043f6990694909ccd86a9060132661bab7ad" exitCode=0 Nov 01 20:35:57.917823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:57.917450 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" event={"ID":"d17871a7-c69c-4a6c-a567-dbe08d346f20","Type":"ContainerDied","Data":"6111bdc2da2278a05ac6674aaac4043f6990694909ccd86a9060132661bab7ad"} Nov 01 20:35:59.077786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.077755 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:35:59.105161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.105137 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:35:59.185522 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.185431 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-openshift-ca\") pod \"d17871a7-c69c-4a6c-a567-dbe08d346f20\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " Nov 01 20:35:59.185705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.185683 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17871a7-c69c-4a6c-a567-dbe08d346f20-kube-api-access-6gsxf\" (UniqueName: \"kubernetes.io/projected/d17871a7-c69c-4a6c-a567-dbe08d346f20-kube-api-access-6gsxf\") pod \"d17871a7-c69c-4a6c-a567-dbe08d346f20\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " Nov 01 20:35:59.185802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.185787 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-xks-ca\") pod \"d17871a7-c69c-4a6c-a567-dbe08d346f20\" (UID: \"d17871a7-c69c-4a6c-a567-dbe08d346f20\") " Nov 01 20:35:59.186097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.186028 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-openshift-ca" pod "d17871a7-c69c-4a6c-a567-dbe08d346f20" (UID: "d17871a7-c69c-4a6c-a567-dbe08d346f20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:35:59.191563 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.191501 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-xks-ca" pod "d17871a7-c69c-4a6c-a567-dbe08d346f20" (UID: "d17871a7-c69c-4a6c-a567-dbe08d346f20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:35:59.194090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.194049 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17871a7-c69c-4a6c-a567-dbe08d346f20-kube-api-access-6gsxf" pod "d17871a7-c69c-4a6c-a567-dbe08d346f20" (UID: "d17871a7-c69c-4a6c-a567-dbe08d346f20"). InnerVolumeSpecName "kube-api-access-6gsxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:59.287358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.287287 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gsxf\" (UniqueName: \"kubernetes.io/projected/d17871a7-c69c-4a6c-a567-dbe08d346f20-kube-api-access-6gsxf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:59.287358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.287319 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:59.287358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.287347 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d17871a7-c69c-4a6c-a567-dbe08d346f20-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:59.994938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.994825 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" event={"ID":"d17871a7-c69c-4a6c-a567-dbe08d346f20","Type":"ContainerDied","Data":"a58036c27cb1a97acc84e192586c6577d9e8bf5daf6a424ec5368bcf52881717"} Nov 01 20:35:59.996710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.996576 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a58036c27cb1a97acc84e192586c6577d9e8bf5daf6a424ec5368bcf52881717" Nov 01 20:35:59.996710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:35:59.995239 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l" Nov 01 20:36:00.574265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:00.574226 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:00.796342 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:00.796290 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:00.923044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:00.922943 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:01.190098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:01.190011 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz"] Nov 01 20:36:01.241734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:01.241698 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37811a-5w6cz"] Nov 01 20:36:01.259750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:01.259712 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568"] Nov 01 20:36:01.260019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:01.259990 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" containerID="cri-o://eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3" gracePeriod=30 Nov 01 20:36:01.289458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:01.289426 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:02.025050 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.025013 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:02.469242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.469215 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695ff339-3f54-45dd-a068-161276bc9b70" path="/var/lib/kubelet/pods/695ff339-3f54-45dd-a068-161276bc9b70/volumes" Nov 01 20:36:02.546764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.546730 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr"] Nov 01 20:36:02.547736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.547711 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d17871a7-c69c-4a6c-a567-dbe08d346f20" containerName="revoke-keys" Nov 01 20:36:02.547811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.547738 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17871a7-c69c-4a6c-a567-dbe08d346f20" containerName="revoke-keys" Nov 01 20:36:02.547921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.547910 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17871a7-c69c-4a6c-a567-dbe08d346f20" containerName="revoke-keys" Nov 01 20:36:02.552382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.552359 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.668592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.668566 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-012e1c-openshift-default-dockercfg-9px4f\"" Nov 01 20:36:02.671021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.670490 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.671611 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-credential-socket\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672100 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-envoy\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672178 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-certs\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672268 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istiod-ca-cert\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672424 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-data\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672442 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-token\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672465 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-socket\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672496 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-podinfo\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.672835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.672536 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p55z\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-kube-api-access-8p55z\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.675163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.675129 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr"] Nov 01 20:36:02.773463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773320 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-data\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773417 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-token\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773443 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-socket\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773467 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-podinfo\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773511 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p55z\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-kube-api-access-8p55z\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773574 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-credential-socket\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773590 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-envoy\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773666 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-certs\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773767 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istiod-ca-cert\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.773815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773800 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-data\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.774145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.773872 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-socket\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.774145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.774100 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-certs\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.774253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.774166 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-credential-socket\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.774437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.774420 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istiod-ca-cert\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.776275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.776244 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-envoy\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.776893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.776868 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-podinfo\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.830017 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.829976 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-token\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.830176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.830051 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p55z\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-kube-api-access-8p55z\") pod \"e2e-api-a-012e1c-openshift-default-78786cfc65-95whr\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:02.903619 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:02.903576 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:03.018450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:03.018032 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:03.398397 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:03.398361 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr"] Nov 01 20:36:03.403146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:03.403111 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:36:03.403251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:03.403185 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:36:03.403251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:03.403231 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:36:04.030156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:04.030111 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" event={"ID":"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca","Type":"ContainerStarted","Data":"ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5"} Nov 01 20:36:04.030426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:04.030175 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" event={"ID":"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca","Type":"ContainerStarted","Data":"d865e26cfc51e6e6fe2bd61f353978d8ebf480154c593e14ce2b261a4d1b79eb"} Nov 01 20:36:04.245462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:04.245405 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podStartSLOduration=2.245388878 podStartE2EDuration="2.245388878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:04.195316197 +0000 UTC m=+1971.869575469" watchObservedRunningTime="2026-11-01 20:36:04.245388878 +0000 UTC m=+1971.919648200" Nov 01 20:36:04.904646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:04.904609 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:04.910198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:04.910170 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:05.039040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:05.039008 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:05.040056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:05.040029 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:36:05.235376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:05.235126 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:06.383521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.383479 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:06.545032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.544967 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:36:06.635818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.635779 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-data\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.635818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.635818 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-credential-socket\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.635860 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-certs\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.635907 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-token\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-token\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.635979 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istiod-ca-cert\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636106 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-data" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:06.636229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636135 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-socket\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636170 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-credential-socket" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:06.636229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636190 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-envoy\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636213 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-certs" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:06.636523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636313 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-kube-api-access-lp28w\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-kube-api-access-lp28w\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636387 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-podinfo\") pod \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\" (UID: \"0e5b127f-1db8-4a60-91c1-7bf056c693a3\") " Nov 01 20:36:06.636523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636492 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-socket" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:06.636523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.636515 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istiod-ca-cert" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:36:06.637234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.637197 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.637234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.637225 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.637425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.637242 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.637425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.637260 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.637425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.637274 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.638645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.638619 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-token" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:06.638780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.638666 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-kube-api-access-lp28w" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "kube-api-access-lp28w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:06.638962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.638946 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-podinfo" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:36:06.638962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.638954 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-envoy" pod "0e5b127f-1db8-4a60-91c1-7bf056c693a3" (UID: "0e5b127f-1db8-4a60-91c1-7bf056c693a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:06.738949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.738907 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp28w\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-kube-api-access-lp28w\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.738949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.738943 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.738949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.738953 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:06.739197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:06.738964 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5b127f-1db8-4a60-91c1-7bf056c693a3-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:07.055186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.055151 5633 generic.go:356] "Generic (PLEG): container finished" podID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerID="eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3" exitCode=0 Nov 01 20:36:07.055413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.055214 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" Nov 01 20:36:07.055413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.055238 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" event={"ID":"0e5b127f-1db8-4a60-91c1-7bf056c693a3","Type":"ContainerDied","Data":"eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3"} Nov 01 20:36:07.055413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.055298 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568" event={"ID":"0e5b127f-1db8-4a60-91c1-7bf056c693a3","Type":"ContainerDied","Data":"a0c0fc95728033fd9601de851e69013a50b96cfdade10e374d31a00d949927c4"} Nov 01 20:36:07.055413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.055321 5633 scope.go:122] "RemoveContainer" containerID="eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3" Nov 01 20:36:07.067048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.067024 5633 scope.go:122] "RemoveContainer" containerID="bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1" Nov 01 20:36:07.077375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.077311 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:07.078538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.078520 5633 scope.go:122] "RemoveContainer" containerID="eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3" Nov 01 20:36:07.078836 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:36:07.078819 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3\": container with ID starting with eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3 not found: ID does not exist" containerID="eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3" Nov 01 20:36:07.078873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.078845 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3"} err="failed to get container status \"eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3\": rpc error: code = NotFound desc = could not find container \"eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3\": container with ID starting with eaebd98244d77b4f623b9930eb6e851b6eac29633e51965f5a7335c43c4601a3 not found: ID does not exist" Nov 01 20:36:07.078873 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.078864 5633 scope.go:122] "RemoveContainer" containerID="bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1" Nov 01 20:36:07.079097 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:36:07.079082 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1\": container with ID starting with bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1 not found: ID does not exist" containerID="bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1" Nov 01 20:36:07.079142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.079101 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1"} err="failed to get container status \"bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1\": rpc error: code = NotFound desc = could not find container \"bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1\": container with ID starting with bfacbc98f2be4d7f4bd521121818d430da65728439b43586eb7b1058593afdc1 not found: ID does not exist" Nov 01 20:36:07.222974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.222940 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568"] Nov 01 20:36:07.272143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.272110 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2f37811a-openshift-default-5cd79968f-l7568"] Nov 01 20:36:07.281834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.281801 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:07.414203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:07.414101 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:08.468527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:08.468491 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" path="/var/lib/kubelet/pods/0e5b127f-1db8-4a60-91c1-7bf056c693a3/volumes" Nov 01 20:36:10.232782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.232746 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:10.415607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.415463 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:10.777974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.777940 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:36:10.856277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.856239 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f"] Nov 01 20:36:10.857492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.857464 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" Nov 01 20:36:10.857671 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.857657 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" Nov 01 20:36:10.857989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.857973 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" Nov 01 20:36:10.858100 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.858086 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" Nov 01 20:36:10.872961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.872919 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:10.901525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.901461 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f"] Nov 01 20:36:10.911230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.911195 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scc8x\" (UniqueName: \"kubernetes.io/projected/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-kube-api-access-scc8x\") pod \"payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:10.911752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.911726 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:10.925436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.925396 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-012e1c\"" Nov 01 20:36:10.925648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:10.925629 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-012e1c-dockercfg-f79d9\"" Nov 01 20:36:11.013874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.013834 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:11.014211 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.014193 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scc8x\" (UniqueName: \"kubernetes.io/projected/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-kube-api-access-scc8x\") pod \"payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:11.015558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.015522 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:11.020102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.020077 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9"] Nov 01 20:36:11.020929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.020889 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" Nov 01 20:36:11.021049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.021038 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5b127f-1db8-4a60-91c1-7bf056c693a3" containerName="istio-proxy" Nov 01 20:36:11.026737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.026306 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.041795 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.041693 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9"] Nov 01 20:36:11.076832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.072961 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scc8x\" (UniqueName: \"kubernetes.io/projected/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-kube-api-access-scc8x\") pod \"payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:11.116088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.116040 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92ba181-03a2-43ca-8830-3f28285dd7ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.116280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.116265 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d956\" (UniqueName: \"kubernetes.io/projected/f92ba181-03a2-43ca-8830-3f28285dd7ca-kube-api-access-4d956\") pod \"payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.198373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.198294 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:11.217572 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.217534 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d956\" (UniqueName: \"kubernetes.io/projected/f92ba181-03a2-43ca-8830-3f28285dd7ca-kube-api-access-4d956\") pod \"payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.217752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.217650 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92ba181-03a2-43ca-8830-3f28285dd7ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.218663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.218632 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92ba181-03a2-43ca-8830-3f28285dd7ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.277556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.277485 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d956\" (UniqueName: \"kubernetes.io/projected/f92ba181-03a2-43ca-8830-3f28285dd7ca-kube-api-access-4d956\") pod \"payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.375606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.375511 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:11.578020 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:36:11.577986 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e8cafb_cbb3_4c40_964a_09a4b1ee72b1.slice/crio-e8546cce1db73451c134f27d7e82f22ffb051265841b48141b1546621d33ebc3 WatchSource:0}: Error finding container e8546cce1db73451c134f27d7e82f22ffb051265841b48141b1546621d33ebc3: Status 404 returned error can't find the container with id e8546cce1db73451c134f27d7e82f22ffb051265841b48141b1546621d33ebc3 Nov 01 20:36:11.578151 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.578124 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f"] Nov 01 20:36:11.789378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:11.787293 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9"] Nov 01 20:36:11.790417 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:36:11.790296 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92ba181_03a2_43ca_8830_3f28285dd7ca.slice/crio-98014ef59bd250f7c3fdfa67f480c1d0541c7741d1a74913aff89963ef7d74f1 WatchSource:0}: Error finding container 98014ef59bd250f7c3fdfa67f480c1d0541c7741d1a74913aff89963ef7d74f1: Status 404 returned error can't find the container with id 98014ef59bd250f7c3fdfa67f480c1d0541c7741d1a74913aff89963ef7d74f1 Nov 01 20:36:12.130251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.130203 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" event={"ID":"f92ba181-03a2-43ca-8830-3f28285dd7ca","Type":"ContainerStarted","Data":"40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd"} Nov 01 20:36:12.130457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.130308 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" event={"ID":"f92ba181-03a2-43ca-8830-3f28285dd7ca","Type":"ContainerStarted","Data":"98014ef59bd250f7c3fdfa67f480c1d0541c7741d1a74913aff89963ef7d74f1"} Nov 01 20:36:12.131504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.131476 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:12.134734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.134707 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" event={"ID":"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1","Type":"ContainerStarted","Data":"8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b"} Nov 01 20:36:12.134952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.134925 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" event={"ID":"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1","Type":"ContainerStarted","Data":"e8546cce1db73451c134f27d7e82f22ffb051265841b48141b1546621d33ebc3"} Nov 01 20:36:12.135095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.135083 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:12.250049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.249982 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" podStartSLOduration=2.249964744 podStartE2EDuration="2.249964744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:12.243852305 +0000 UTC m=+1979.918111602" watchObservedRunningTime="2026-11-01 20:36:12.249964744 +0000 UTC m=+1979.924223999" Nov 01 20:36:12.367185 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.365386 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" podStartSLOduration=2.3653605300000002 podStartE2EDuration="2.36536053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:12.356828613 +0000 UTC m=+1980.031087872" watchObservedRunningTime="2026-11-01 20:36:12.36536053 +0000 UTC m=+1980.039619789" Nov 01 20:36:12.513510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.513408 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:12.681834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:12.681780 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:13.647362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:13.642940 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:13.941154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:13.941044 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:14.744521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:14.744472 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:36:14.928509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:14.928465 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:36:14.928850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:14.928816 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:36:15.055577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:15.055443 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:36:15.055844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:15.055804 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:36:15.090527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:15.090447 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:36:15.322419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:15.317962 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:15.966883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:15.966765 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:16.131203 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.131159 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d"] Nov 01 20:36:16.131751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.131714 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" containerID="cri-o://9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0" gracePeriod=30 Nov 01 20:36:16.198016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.197649 5633 patch_prober.go:29] interesting pod/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:36:16.198016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.197729 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:36:16.452115 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:36:16.437988 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:36:16.452115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.438937 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:16.863657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.863610 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5"] Nov 01 20:36:16.868379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.868317 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:16.935094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.923509 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-912d17-openshift-default-dockercfg-wwjt2\"" Nov 01 20:36:16.960420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:16.959310 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:17.010757 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.010720 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-socket\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.012743 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.012706 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-token\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.013201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.013175 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hl64\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-kube-api-access-8hl64\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.013361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.013231 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8768f69a-0d62-4fd6-89d7-515530ffab12-istiod-ca-cert\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.013361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.013306 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-data\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.013482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.013438 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-credential-socket\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.013547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.013497 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-envoy\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.013604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.013544 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-certs\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.013604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.013572 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-podinfo\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.019384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.017278 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5"] Nov 01 20:36:17.025121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.025080 5633 scope.go:122] "RemoveContainer" containerID="666f5ae2e98249e6de1c72343d84632012a884df14e24425dc8f46e2882ba358" Nov 01 20:36:17.087083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.079616 5633 scope.go:122] "RemoveContainer" containerID="9c277a755571bac546b951e4197838a0d9e7630f70df912d215105809715ac56" Nov 01 20:36:17.114796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.114713 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-socket\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.114796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.114763 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-token\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.115061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.114851 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hl64\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-kube-api-access-8hl64\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.115061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.114891 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8768f69a-0d62-4fd6-89d7-515530ffab12-istiod-ca-cert\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.115061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.114956 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-data\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.115239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.115064 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-credential-socket\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.115239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.115119 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-envoy\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.115239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.115161 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-certs\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.115239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.115188 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-podinfo\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.117108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.117008 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8768f69a-0d62-4fd6-89d7-515530ffab12-istiod-ca-cert\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.117255 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.117236 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-credential-socket\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.117588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.117559 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-data\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.117850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.117827 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-socket\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.117954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.117860 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-certs\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.119361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.119113 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-podinfo\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.120163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.120139 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-envoy\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.121578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.121550 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:17.173947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.173919 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-token\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.174245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.174023 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hl64\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-kube-api-access-8hl64\") pod \"e2e-noten-912d17-openshift-default-7d6fc8797-d85j5\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.184655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.184619 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:17.627475 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.627433 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5"] Nov 01 20:36:17.664610 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:36:17.664578 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8768f69a_0d62_4fd6_89d7_515530ffab12.slice/crio-00558dec3955ddba2dd7246819f4635194f13e84eaf74d34df6ebe3a90917c0f WatchSource:0}: Error finding container 00558dec3955ddba2dd7246819f4635194f13e84eaf74d34df6ebe3a90917c0f: Status 404 returned error can't find the container with id 00558dec3955ddba2dd7246819f4635194f13e84eaf74d34df6ebe3a90917c0f Nov 01 20:36:17.667362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.667292 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:36:17.667456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.667398 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:36:17.667456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:17.667435 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:36:18.035304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:18.035265 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:18.473039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:18.473000 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" event={"ID":"8768f69a-0d62-4fd6-89d7-515530ffab12","Type":"ContainerStarted","Data":"65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9"} Nov 01 20:36:18.473223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:18.473059 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" event={"ID":"8768f69a-0d62-4fd6-89d7-515530ffab12","Type":"ContainerStarted","Data":"00558dec3955ddba2dd7246819f4635194f13e84eaf74d34df6ebe3a90917c0f"} Nov 01 20:36:18.688879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:18.688806 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" podStartSLOduration=2.688786377 podStartE2EDuration="2.688786377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:18.637360279 +0000 UTC m=+1986.311619535" watchObservedRunningTime="2026-11-01 20:36:18.688786377 +0000 UTC m=+1986.363045633" Nov 01 20:36:19.005177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:19.005132 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:19.181359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:19.180593 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:19.185426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:19.185053 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:19.209500 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:19.209447 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:19.458002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:19.457918 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:19.481425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:19.481378 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:19.482489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:19.482466 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:36:20.449756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.449721 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn"] Nov 01 20:36:20.454277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.454252 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.460621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.460590 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-plugins-config-volume\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.460788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.460759 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-routing-overlay-volume\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.460836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.460803 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-provider-credentials\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.461379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.461059 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5bl\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-kube-api-access-wt5bl\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.529213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.529165 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn"] Nov 01 20:36:20.562553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.562514 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-routing-overlay-volume\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.562744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.562682 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-provider-credentials\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.562917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.562898 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5bl\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-kube-api-access-wt5bl\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.562993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.562962 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-plugins-config-volume\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.563218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.563188 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-routing-overlay-volume\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.563664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.563640 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-plugins-config-volume\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.565744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.565716 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-provider-credentials\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.619740 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.619696 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5bl\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-kube-api-access-wt5bl\") pod \"payload-processing-external-model-78bf6d8768-gtpqn\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:20.773042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:20.773001 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:21.156542 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:36:21.156494 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b81c8c6_5ed3_4628_9f11_8e4815004937.slice/crio-5c06ee581205bae941e438e04f367ab6ac89e483cf208ee06f1aa61236bf01a7 WatchSource:0}: Error finding container 5c06ee581205bae941e438e04f367ab6ac89e483cf208ee06f1aa61236bf01a7: Status 404 returned error can't find the container with id 5c06ee581205bae941e438e04f367ab6ac89e483cf208ee06f1aa61236bf01a7 Nov 01 20:36:21.157408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.157362 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn"] Nov 01 20:36:21.420201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.420172 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:36:21.475482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475454 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istiod-ca-cert\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.475482 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475489 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-socket\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475597 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-envoy\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475625 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-credential-socket\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475650 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-podinfo\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475715 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-kube-api-access-hxh5f\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-kube-api-access-hxh5f\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475760 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-certs\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475870 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istiod-ca-cert" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475873 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-socket" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:21.476069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475917 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-credential-socket" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:21.476713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476087 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-certs" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:21.476713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.475862 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-token\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-token\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476210 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-data\") pod \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\" (UID: \"d24418ce-cbc8-414a-92f7-9f2ed7f338b3\") " Nov 01 20:36:21.476713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476455 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-data" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:21.476956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476816 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.476956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476835 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.476956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476849 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.476956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476857 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.476956 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.476866 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.478471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.478432 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-token" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:21.478599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.478538 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-podinfo" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:36:21.478599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.478574 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-kube-api-access-hxh5f" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "kube-api-access-hxh5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:21.478851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.478828 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-envoy" pod "d24418ce-cbc8-414a-92f7-9f2ed7f338b3" (UID: "d24418ce-cbc8-414a-92f7-9f2ed7f338b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:21.497217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.497169 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" event={"ID":"9b81c8c6-5ed3-4628-9f11-8e4815004937","Type":"ContainerStarted","Data":"9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870"} Nov 01 20:36:21.497422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.497247 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:36:21.497422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.497271 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" event={"ID":"9b81c8c6-5ed3-4628-9f11-8e4815004937","Type":"ContainerStarted","Data":"5c06ee581205bae941e438e04f367ab6ac89e483cf208ee06f1aa61236bf01a7"} Nov 01 20:36:21.499364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.499316 5633 generic.go:356] "Generic (PLEG): container finished" podID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerID="9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0" exitCode=0 Nov 01 20:36:21.499509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.499379 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" event={"ID":"d24418ce-cbc8-414a-92f7-9f2ed7f338b3","Type":"ContainerDied","Data":"9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0"} Nov 01 20:36:21.499509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.499428 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" event={"ID":"d24418ce-cbc8-414a-92f7-9f2ed7f338b3","Type":"ContainerDied","Data":"feab7b2587b3e86068267b72ee3bf23e54fd8aa60ff1c3e31da07fe30f021238"} Nov 01 20:36:21.499509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.499449 5633 scope.go:122] "RemoveContainer" containerID="9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0" Nov 01 20:36:21.499509 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.499386 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d" Nov 01 20:36:21.510598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.510576 5633 scope.go:122] "RemoveContainer" containerID="f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735" Nov 01 20:36:21.520913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.520894 5633 scope.go:122] "RemoveContainer" containerID="9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0" Nov 01 20:36:21.521237 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:36:21.521216 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0\": container with ID starting with 9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0 not found: ID does not exist" containerID="9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0" Nov 01 20:36:21.521315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.521250 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0"} err="failed to get container status \"9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0\": rpc error: code = NotFound desc = could not find container \"9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0\": container with ID starting with 9228599261aa77c9d8d4d5b20cc33c4c45c3849eb4e3549910020b99328682a0 not found: ID does not exist" Nov 01 20:36:21.521315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.521277 5633 scope.go:122] "RemoveContainer" containerID="f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735" Nov 01 20:36:21.521579 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:36:21.521558 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735\": container with ID starting with f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735 not found: ID does not exist" containerID="f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735" Nov 01 20:36:21.521616 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.521585 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735"} err="failed to get container status \"f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735\": rpc error: code = NotFound desc = could not find container \"f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735\": container with ID starting with f0ef90c61acbb6f816e20a072efb37b1b3176afdcb6195758ece903932087735 not found: ID does not exist" Nov 01 20:36:21.581003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.580970 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.581147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.581009 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.581147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.581026 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxh5f\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-kube-api-access-hxh5f\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.581147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.581040 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24418ce-cbc8-414a-92f7-9f2ed7f338b3-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:21.650125 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.650087 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d"] Nov 01 20:36:21.654015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.653991 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-f154f7-openshift-default-5d6b8d5965-59k7d"] Nov 01 20:36:21.762250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:21.762189 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podStartSLOduration=1.7621741260000001 podStartE2EDuration="1.762174126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:21.7603824 +0000 UTC m=+1989.434641657" watchObservedRunningTime="2026-11-01 20:36:21.762174126 +0000 UTC m=+1989.436433434" Nov 01 20:36:22.321277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:22.321240 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:22.468605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:22.468561 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" path="/var/lib/kubelet/pods/d24418ce-cbc8-414a-92f7-9f2ed7f338b3/volumes" Nov 01 20:36:22.901111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:22.901075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:23.150938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:23.150909 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:36:24.269682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:24.269654 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:36:25.428494 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:25.428457 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:25.604772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:25.604739 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:27.577630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:27.577592 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:28.782851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:28.782805 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:32.515625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:32.515589 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:32.555555 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:32.555509 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:36:37.325825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:37.325787 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:41.932392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:41.932313 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:42.090318 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:42.090276 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:42.206031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:42.205928 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:42.509886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:42.509842 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:36:46.161394 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:36:46.161350 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:36:46.786760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:46.786729 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:51.504441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:51.504394 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:52.510201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:52.510159 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:36:56.175088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:56.175045 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:56.479823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:56.479778 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:36:59.313404 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:59.313369 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:59.473786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:59.473745 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:59.476895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:59.476862 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5"] Nov 01 20:36:59.477402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:59.477370 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" podUID="8768f69a-0d62-4fd6-89d7-515530ffab12" containerName="istio-proxy" containerID="cri-o://65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9" gracePeriod=30 Nov 01 20:36:59.774241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:59.774205 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:36:59.912934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:36:59.912898 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:01.028595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:01.028552 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:02.511280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:02.511236 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:37:04.484732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.484689 5633 patch_prober.go:29] interesting pod/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:37:04.485166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.484770 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" podUID="8768f69a-0d62-4fd6-89d7-515530ffab12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:37:04.781663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.781626 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:37:04.871629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.871590 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-envoy\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.871808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.871644 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-podinfo\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.871808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.871693 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-data\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.871808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.871787 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-kube-api-access-8hl64\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-kube-api-access-8hl64\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.871994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.871848 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8768f69a-0d62-4fd6-89d7-515530ffab12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8768f69a-0d62-4fd6-89d7-515530ffab12-istiod-ca-cert\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.871994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.871909 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-credential-socket\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.872107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872006 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-token\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-token\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.872107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872049 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-certs\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.872230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872157 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-socket\") pod \"8768f69a-0d62-4fd6-89d7-515530ffab12\" (UID: \"8768f69a-0d62-4fd6-89d7-515530ffab12\") " Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872043 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-data" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872215 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-credential-socket" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872460 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-certs" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872279 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8768f69a-0d62-4fd6-89d7-515530ffab12-istiod-ca-cert" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872639 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-socket" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872759 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872778 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872788 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872797 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8768f69a-0d62-4fd6-89d7-515530ffab12-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.872839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.872806 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.874710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.874681 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-envoy" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:04.874841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.874749 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-podinfo" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:37:04.874906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.874883 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-kube-api-access-8hl64" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "kube-api-access-8hl64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:04.874962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.874905 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-token" pod "8768f69a-0d62-4fd6-89d7-515530ffab12" (UID: "8768f69a-0d62-4fd6-89d7-515530ffab12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:04.950437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.950396 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:04.974090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.974056 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hl64\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-kube-api-access-8hl64\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.974090 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.974091 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.974394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.974107 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.974394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.974119 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8768f69a-0d62-4fd6-89d7-515530ffab12-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:04.980360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.980297 5633 generic.go:356] "Generic (PLEG): container finished" podID="8768f69a-0d62-4fd6-89d7-515530ffab12" containerID="65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9" exitCode=0 Nov 01 20:37:04.980545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.980398 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" Nov 01 20:37:04.980545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.980375 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" event={"ID":"8768f69a-0d62-4fd6-89d7-515530ffab12","Type":"ContainerDied","Data":"65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9"} Nov 01 20:37:04.980677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.980556 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5" event={"ID":"8768f69a-0d62-4fd6-89d7-515530ffab12","Type":"ContainerDied","Data":"00558dec3955ddba2dd7246819f4635194f13e84eaf74d34df6ebe3a90917c0f"} Nov 01 20:37:04.980677 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:04.980592 5633 scope.go:122] "RemoveContainer" containerID="65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9" Nov 01 20:37:05.001889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:05.001863 5633 scope.go:122] "RemoveContainer" containerID="65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9" Nov 01 20:37:05.002425 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:05.002348 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9\": container with ID starting with 65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9 not found: ID does not exist" containerID="65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9" Nov 01 20:37:05.002425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:05.002395 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9"} err="failed to get container status \"65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9\": rpc error: code = NotFound desc = could not find container \"65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9\": container with ID starting with 65aef232bb817de6afdcb130fc74b4865941da7bc18aac6cd6b73d1b42a8f2d9 not found: ID does not exist" Nov 01 20:37:05.146626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:05.146586 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5"] Nov 01 20:37:05.150709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:05.150685 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-912d17-openshift-default-7d6fc8797-d85j5"] Nov 01 20:37:05.812264 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:05.812225 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:06.080552 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:06.080466 5633 patch_prober.go:29] interesting pod/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:37:06.080704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:06.080555 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:37:06.468168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:06.468135 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8768f69a-0d62-4fd6-89d7-515530ffab12" path="/var/lib/kubelet/pods/8768f69a-0d62-4fd6-89d7-515530ffab12/volumes" Nov 01 20:37:06.998451 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:06.998366 5633 generic.go:356] "Generic (PLEG): container finished" podID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerID="ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5" exitCode=0 Nov 01 20:37:06.998909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:06.998448 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" event={"ID":"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca","Type":"ContainerDied","Data":"ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5"} Nov 01 20:37:06.998909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:06.998878 5633 scope.go:122] "RemoveContainer" containerID="ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5" Nov 01 20:37:07.000315 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:07.000248 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:37:07.000456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:07.000383 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:37:07.000456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:07.000431 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:37:07.712123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:07.712085 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:07.844755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:07.844720 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:08.022120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:08.022038 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" event={"ID":"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca","Type":"ContainerStarted","Data":"05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2"} Nov 01 20:37:08.910355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:08.909766 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:37:08.910355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:08.909901 5633 patch_prober.go:29] interesting pod/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 20:37:08.910355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:08.909944 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 20:37:09.906808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:09.906771 5633 patch_prober.go:29] interesting pod/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 20:37:09.907459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:09.907421 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 20:37:10.622523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:10.622479 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:10.907454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:10.907355 5633 patch_prober.go:29] interesting pod/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 20:37:10.908049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:10.908013 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 20:37:11.904838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:11.904801 5633 patch_prober.go:29] interesting pod/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 20:37:11.905147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:11.905118 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 20:37:12.511984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:12.511945 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:37:12.909173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:12.909076 5633 patch_prober.go:29] interesting pod/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 20:37:12.909508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:12.909462 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 20:37:13.908150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:13.908114 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:37:13.908665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:13.908601 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:37:13.909248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:13.909225 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:37:14.502096 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:14.502061 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4ca90c301433: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:14.502374 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:14.502354 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:14.502592 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:14.502569 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:37:15.002546139 +0000 UTC m=+2042.676805424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-4ca90c301433" not found Nov 01 20:37:15.009463 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:15.009430 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4ca90c301433: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:15.009463 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:15.009458 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:15.009910 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:15.009510 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:37:16.009496587 +0000 UTC m=+2043.683755820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-4ca90c301433" not found Nov 01 20:37:15.375608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:15.375527 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:16.023884 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:16.023853 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4ca90c301433: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:16.023884 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:16.023883 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:16.024346 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:16.023939 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:37:18.023920361 +0000 UTC m=+2045.698179596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-4ca90c301433" not found Nov 01 20:37:16.158122 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:16.158075 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:37:17.187367 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:17.187297 5633 scope.go:122] "RemoveContainer" containerID="30140ae1d3f72bfca04d0f226b6ca97e0d131e383d4002dc0ece15c881fd55c2" Nov 01 20:37:17.198209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:17.198181 5633 scope.go:122] "RemoveContainer" containerID="69ef635891b0a504ab6fb7a983995b90da7795a34d9382ca33891d946bdcd65c" Nov 01 20:37:17.208507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:17.208482 5633 scope.go:122] "RemoveContainer" containerID="0c89f15c83e4719952dd07d1dba2414f7163ee13ad106e27891cdc3d0da9b3ba" Nov 01 20:37:18.050173 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:18.050132 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4ca90c301433: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:18.050173 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:18.050163 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-4ca90c301433" not found Nov 01 20:37:18.050460 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:18.050231 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:37:22.050217346 +0000 UTC m=+2049.724476580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-4ca90c301433" not found Nov 01 20:37:20.086523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:20.086477 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:22.118385 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:22.118321 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b8ed0bcb9390: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:22.118385 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:22.118378 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:22.118947 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:22.118449 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:37:30.118426788 +0000 UTC m=+2057.792686027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:22.510171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:22.510132 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:37:24.232673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.232628 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn"] Nov 01 20:37:24.235280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.235223 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:37:24.260642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.260602 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w"] Nov 01 20:37:24.262604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.262572 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" Nov 01 20:37:24.262604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.262607 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" Nov 01 20:37:24.262852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.262637 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" Nov 01 20:37:24.262852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.262645 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" Nov 01 20:37:24.262852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.262681 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8768f69a-0d62-4fd6-89d7-515530ffab12" containerName="istio-proxy" Nov 01 20:37:24.262852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.262695 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8768f69a-0d62-4fd6-89d7-515530ffab12" containerName="istio-proxy" Nov 01 20:37:24.263055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.263012 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" Nov 01 20:37:24.263055 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.263035 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24418ce-cbc8-414a-92f7-9f2ed7f338b3" containerName="istio-proxy" Nov 01 20:37:24.263145 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.263055 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8768f69a-0d62-4fd6-89d7-515530ffab12" containerName="istio-proxy" Nov 01 20:37:24.273624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.270820 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.344578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.344540 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w"] Nov 01 20:37:24.379724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.379666 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-routing-overlay-volume\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.379948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.379889 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg66b\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-kube-api-access-gg66b\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.380138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.380103 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-provider-credentials\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.380275 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.380173 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-plugins-config-volume\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.482153 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.482115 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-provider-credentials\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.482431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.482172 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-plugins-config-volume\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.482431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.482271 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-routing-overlay-volume\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.482576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.482440 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg66b\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-kube-api-access-gg66b\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.483041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.482996 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-routing-overlay-volume\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.483041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.483015 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-plugins-config-volume\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.484840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.484819 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-provider-credentials\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.538606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.538576 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg66b\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-kube-api-access-gg66b\") pod \"payload-processing-external-model-65b97f6f5b-hgf8w\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.590538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.590507 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:24.891790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.891705 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:24.975393 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:37:24.971410 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30293fac_754e_4991_98d4_3ca8397d81ad.slice/crio-493d715df428a886980e9759960acc532d48302e734b87e505cee239144466af WatchSource:0}: Error finding container 493d715df428a886980e9759960acc532d48302e734b87e505cee239144466af: Status 404 returned error can't find the container with id 493d715df428a886980e9759960acc532d48302e734b87e505cee239144466af Nov 01 20:37:24.975393 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:24.971573 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w"] Nov 01 20:37:25.204543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:25.204455 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" event={"ID":"30293fac-754e-4991-98d4-3ca8397d81ad","Type":"ContainerStarted","Data":"c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440"} Nov 01 20:37:25.204543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:25.204526 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" event={"ID":"30293fac-754e-4991-98d4-3ca8397d81ad","Type":"ContainerStarted","Data":"493d715df428a886980e9759960acc532d48302e734b87e505cee239144466af"} Nov 01 20:37:25.205383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:25.205364 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:37:25.318444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:25.318388 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podStartSLOduration=1.3183716460000001 podStartE2EDuration="1.318371646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:25.31714261 +0000 UTC m=+2052.991401866" watchObservedRunningTime="2026-11-01 20:37:25.318371646 +0000 UTC m=+2052.992630942" Nov 01 20:37:28.338583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:28.338531 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:29.160575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.160535 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:29.234323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.234239 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" containerID="cri-o://9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870" gracePeriod=25 Nov 01 20:37:29.308358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.307083 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:29.784359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.781562 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:37:29.808360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.798895 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6"] Nov 01 20:37:29.808360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.799893 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" Nov 01 20:37:29.808360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.799917 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" Nov 01 20:37:29.808360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.800104 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerName="payload-processing" Nov 01 20:37:29.812357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.810638 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:29.847456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.840881 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:29.900106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.899141 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-13cbc9-openshift-default-dockercfg-m4mdz\"" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.941532 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-provider-credentials\") pod \"9b81c8c6-5ed3-4628-9f11-8e4815004937\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.941618 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-kube-api-access-wt5bl\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-kube-api-access-wt5bl\") pod \"9b81c8c6-5ed3-4628-9f11-8e4815004937\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.941688 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-plugins-config-volume\") pod \"9b81c8c6-5ed3-4628-9f11-8e4815004937\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.941781 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-routing-overlay-volume\") pod \"9b81c8c6-5ed3-4628-9f11-8e4815004937\" (UID: \"9b81c8c6-5ed3-4628-9f11-8e4815004937\") " Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942079 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-socket\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942111 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e7055e-4677-478f-bb8e-4423b79d72c0-istiod-ca-cert\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942219 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-data\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942398 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-envoy\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942461 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-token\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942522 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8bp8\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-kube-api-access-x8bp8\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942570 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-podinfo\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942671 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-credential-socket\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.015357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:29.942721 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-certs\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.100296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.073078 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-routing-overlay-volume" pod "9b81c8c6-5ed3-4628-9f11-8e4815004937" (UID: "9b81c8c6-5ed3-4628-9f11-8e4815004937"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:37:30.100296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.073464 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-plugins-config-volume" pod "9b81c8c6-5ed3-4628-9f11-8e4815004937" (UID: "9b81c8c6-5ed3-4628-9f11-8e4815004937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:37:30.100296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.076105 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-envoy\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.100296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.076149 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-token\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.100296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.076207 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8bp8\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-kube-api-access-x8bp8\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.100296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.076253 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-podinfo\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.111472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.111433 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-credential-socket\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.111763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.111740 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-certs\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.137364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.111925 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-socket\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.137819 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.137787 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e7055e-4677-478f-bb8e-4423b79d72c0-istiod-ca-cert\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.138118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.138097 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-data\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.138519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.138499 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:30.138665 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.138649 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b81c8c6-5ed3-4628-9f11-8e4815004937-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:30.138776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.120017 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-socket\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.138909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.119552 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-credential-socket\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.161647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.139775 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e7055e-4677-478f-bb8e-4423b79d72c0-istiod-ca-cert\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.207358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.119793 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-certs\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.207358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.153125 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-data\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.207358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.190319 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6"] Nov 01 20:37:30.208219 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.207904 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:30.258357 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:30.252381 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b8ed0bcb9390: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:30.258357 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:30.252419 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:30.258357 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:30.252492 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:37:46.252470802 +0000 UTC m=+2073.926730045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:30.429033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.415693 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:30.456576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.456529 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8bp8\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-kube-api-access-x8bp8\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.531361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.528651 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-provider-credentials" pod "9b81c8c6-5ed3-4628-9f11-8e4815004937" (UID: "9b81c8c6-5ed3-4628-9f11-8e4815004937"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:30.531361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.530519 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-kube-api-access-wt5bl" pod "9b81c8c6-5ed3-4628-9f11-8e4815004937" (UID: "9b81c8c6-5ed3-4628-9f11-8e4815004937"). InnerVolumeSpecName "kube-api-access-wt5bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:30.531361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.531101 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-podinfo\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.532123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.532098 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-envoy\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.574323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.553908 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-token\") pod \"e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.623473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.612873 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:30.623473 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.612927 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt5bl\" (UniqueName: \"kubernetes.io/projected/9b81c8c6-5ed3-4628-9f11-8e4815004937-kube-api-access-wt5bl\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:30.668946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.668888 5633 generic.go:356] "Generic (PLEG): container finished" podID="9b81c8c6-5ed3-4628-9f11-8e4815004937" containerID="9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870" exitCode=0 Nov 01 20:37:30.694978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.672492 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" Nov 01 20:37:30.695436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.672519 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" event={"ID":"9b81c8c6-5ed3-4628-9f11-8e4815004937","Type":"ContainerDied","Data":"9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870"} Nov 01 20:37:30.733071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.731511 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn" event={"ID":"9b81c8c6-5ed3-4628-9f11-8e4815004937","Type":"ContainerDied","Data":"5c06ee581205bae941e438e04f367ab6ac89e483cf208ee06f1aa61236bf01a7"} Nov 01 20:37:30.733071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.731572 5633 scope.go:122] "RemoveContainer" containerID="9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870" Nov 01 20:37:30.790664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.790618 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:30.850234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.850207 5633 scope.go:122] "RemoveContainer" containerID="9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870" Nov 01 20:37:30.850968 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:30.850941 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870\": container with ID starting with 9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870 not found: ID does not exist" containerID="9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870" Nov 01 20:37:30.851143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.851115 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870"} err="failed to get container status \"9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870\": rpc error: code = NotFound desc = could not find container \"9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870\": container with ID starting with 9413c5be65fb44e5811d5d1a6aa3f005b0d7893adf62ed1f67ebe3a5e5ea8870 not found: ID does not exist" Nov 01 20:37:30.973899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:30.965170 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn"] Nov 01 20:37:31.030372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:31.019429 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf6d8768-gtpqn"] Nov 01 20:37:31.581686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:31.581648 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6"] Nov 01 20:37:31.631563 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:37:31.631501 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e7055e_4677_478f_bb8e_4423b79d72c0.slice/crio-90a2035e66dd25e5a997a753771dd5c9218d6f398e2a9a2dc2621465308cc0aa WatchSource:0}: Error finding container 90a2035e66dd25e5a997a753771dd5c9218d6f398e2a9a2dc2621465308cc0aa: Status 404 returned error can't find the container with id 90a2035e66dd25e5a997a753771dd5c9218d6f398e2a9a2dc2621465308cc0aa Nov 01 20:37:31.673174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:31.661948 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:37:31.673174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:31.662083 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:37:31.673174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:31.662131 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:37:31.892213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:31.892127 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" event={"ID":"76e7055e-4677-478f-bb8e-4423b79d72c0","Type":"ContainerStarted","Data":"90a2035e66dd25e5a997a753771dd5c9218d6f398e2a9a2dc2621465308cc0aa"} Nov 01 20:37:32.592356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:32.587932 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b81c8c6-5ed3-4628-9f11-8e4815004937" path="/var/lib/kubelet/pods/9b81c8c6-5ed3-4628-9f11-8e4815004937/volumes" Nov 01 20:37:32.940435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:32.928147 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" event={"ID":"76e7055e-4677-478f-bb8e-4423b79d72c0","Type":"ContainerStarted","Data":"fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7"} Nov 01 20:37:33.792280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:33.792235 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:33.798319 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:33.798288 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:33.936159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:33.936123 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:33.937180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:33.937157 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:37:33.969276 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:33.969215 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" podStartSLOduration=4.969199631 podStartE2EDuration="4.969199631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:33.169195736 +0000 UTC m=+2060.843454992" watchObservedRunningTime="2026-11-01 20:37:33.969199631 +0000 UTC m=+2061.643458885" Nov 01 20:37:35.235396 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:35.235362 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:35.934161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:35.934124 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:36.110950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:36.110910 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:36.244148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:36.244114 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:36.836881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:36.836848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:36.979877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:36.979844 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:37.223018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:37.222977 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:37:39.115766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.115719 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:39.355591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.355552 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:39.585804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.585771 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr"] Nov 01 20:37:39.590200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.590184 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:39.683260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.683219 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr"] Nov 01 20:37:39.709453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.709320 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-13cbc9\"" Nov 01 20:37:39.710770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.710747 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-13cbc9-dockercfg-5lgs6\"" Nov 01 20:37:39.761372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.760090 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv"] Nov 01 20:37:39.761372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.760692 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdmt\" (UniqueName: \"kubernetes.io/projected/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-kube-api-access-2wdmt\") pod \"payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:39.761372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.760859 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:39.764824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.764805 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:39.813850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.813814 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv"] Nov 01 20:37:39.862142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.862045 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:39.862351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.862160 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v875s\" (UniqueName: \"kubernetes.io/projected/20615658-7a77-4d68-b8e6-5720a7a1f6aa-kube-api-access-v875s\") pod \"payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:39.862351 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.862231 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdmt\" (UniqueName: \"kubernetes.io/projected/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-kube-api-access-2wdmt\") pod \"payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:39.862498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.862378 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20615658-7a77-4d68-b8e6-5720a7a1f6aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:39.862831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.862810 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:39.919053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.919011 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdmt\" (UniqueName: \"kubernetes.io/projected/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-kube-api-access-2wdmt\") pod \"payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:39.964246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.964211 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20615658-7a77-4d68-b8e6-5720a7a1f6aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:39.964476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.964459 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v875s\" (UniqueName: \"kubernetes.io/projected/20615658-7a77-4d68-b8e6-5720a7a1f6aa-kube-api-access-v875s\") pod \"payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:39.964916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:39.964897 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20615658-7a77-4d68-b8e6-5720a7a1f6aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:40.021456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.021423 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v875s\" (UniqueName: \"kubernetes.io/projected/20615658-7a77-4d68-b8e6-5720a7a1f6aa-kube-api-access-v875s\") pod \"payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:40.086688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.086654 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:40.204930 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.204572 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:40.396709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.396666 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:40.482504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.482475 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv"] Nov 01 20:37:40.603906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.603879 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr"] Nov 01 20:37:40.604453 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:37:40.604427 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52b3ede_52ee_460a_a7c1_c86e0389f4d4.slice/crio-dea33223d6a5fb41ab0a1b74de9a59b7ba8b58db307d57f7fcfc70224b3010bd WatchSource:0}: Error finding container dea33223d6a5fb41ab0a1b74de9a59b7ba8b58db307d57f7fcfc70224b3010bd: Status 404 returned error can't find the container with id dea33223d6a5fb41ab0a1b74de9a59b7ba8b58db307d57f7fcfc70224b3010bd Nov 01 20:37:40.991267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.991225 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" event={"ID":"20615658-7a77-4d68-b8e6-5720a7a1f6aa","Type":"ContainerStarted","Data":"8cead925db64d72eaabb20bf14a9aeb90d593ef75e2e37efdb36436284ce5fb5"} Nov 01 20:37:40.991508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.991295 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:40.991508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.991319 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" event={"ID":"20615658-7a77-4d68-b8e6-5720a7a1f6aa","Type":"ContainerStarted","Data":"d8650bc544701ce5aaf3689cecdf51d0b059a5ff17ff8cde1d54dcfc6c5b5dc1"} Nov 01 20:37:40.993098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.993069 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" event={"ID":"d52b3ede-52ee-460a-a7c1-c86e0389f4d4","Type":"ContainerStarted","Data":"d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22"} Nov 01 20:37:40.993230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.993116 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" event={"ID":"d52b3ede-52ee-460a-a7c1-c86e0389f4d4","Type":"ContainerStarted","Data":"dea33223d6a5fb41ab0a1b74de9a59b7ba8b58db307d57f7fcfc70224b3010bd"} Nov 01 20:37:40.993230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:40.993142 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:41.106759 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:41.106699 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" podStartSLOduration=2.106683606 podStartE2EDuration="2.106683606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:41.103429336 +0000 UTC m=+2068.777688590" watchObservedRunningTime="2026-11-01 20:37:41.106683606 +0000 UTC m=+2068.780942861" Nov 01 20:37:41.224545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:41.224175 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" podStartSLOduration=2.224150623 podStartE2EDuration="2.224150623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:41.216918661 +0000 UTC m=+2068.891177917" watchObservedRunningTime="2026-11-01 20:37:41.224150623 +0000 UTC m=+2068.898409879" Nov 01 20:37:41.597181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:41.597146 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:41.773666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:41.773632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:42.316685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.316652 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7"] Nov 01 20:37:42.321120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.321099 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.376751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.376718 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7"] Nov 01 20:37:42.420411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.420365 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:37:42.420574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.420422 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:37:42.420574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.420432 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:37:42.420574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.420467 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:37:42.502721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.502675 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vmm\" (UniqueName: \"kubernetes.io/projected/0a799591-35e2-4292-8f87-b1f7b6764aee-kube-api-access-g9vmm\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.502921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.502741 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.502921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.502905 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.604137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.604041 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vmm\" (UniqueName: \"kubernetes.io/projected/0a799591-35e2-4292-8f87-b1f7b6764aee-kube-api-access-g9vmm\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.604137 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.604082 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.604425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.604164 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.604888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.604869 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.606943 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.606922 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.662753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.662722 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vmm\" (UniqueName: \"kubernetes.io/projected/0a799591-35e2-4292-8f87-b1f7b6764aee-kube-api-access-g9vmm\") pod \"maas-api-revoke-sub-289dd985-nzpj7\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:42.936357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:42.936253 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:43.320122 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:37:43.320087 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a799591_35e2_4292_8f87_b1f7b6764aee.slice/crio-814e7e27f0f7d500e92a6d962bdc3d19d4c04807beafd716261e5f34bed0ab0a WatchSource:0}: Error finding container 814e7e27f0f7d500e92a6d962bdc3d19d4c04807beafd716261e5f34bed0ab0a: Status 404 returned error can't find the container with id 814e7e27f0f7d500e92a6d962bdc3d19d4c04807beafd716261e5f34bed0ab0a Nov 01 20:37:43.320642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:43.320618 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7"] Nov 01 20:37:44.027486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:44.027446 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" event={"ID":"0a799591-35e2-4292-8f87-b1f7b6764aee","Type":"ContainerStarted","Data":"ecd0be992badf00e36d1ac55ad310ed7914a069bf6f52ab7c0a53e095e33e169"} Nov 01 20:37:44.027699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:44.027501 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" event={"ID":"0a799591-35e2-4292-8f87-b1f7b6764aee","Type":"ContainerStarted","Data":"814e7e27f0f7d500e92a6d962bdc3d19d4c04807beafd716261e5f34bed0ab0a"} Nov 01 20:37:44.143348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:44.143286 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" podStartSLOduration=2.143271748 podStartE2EDuration="2.143271748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:37:44.139364377 +0000 UTC m=+2071.813623633" watchObservedRunningTime="2026-11-01 20:37:44.143271748 +0000 UTC m=+2071.817531003" Nov 01 20:37:45.037135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:45.037088 5633 generic.go:356] "Generic (PLEG): container finished" podID="0a799591-35e2-4292-8f87-b1f7b6764aee" containerID="ecd0be992badf00e36d1ac55ad310ed7914a069bf6f52ab7c0a53e095e33e169" exitCode=0 Nov 01 20:37:45.037758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:45.037191 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" event={"ID":"0a799591-35e2-4292-8f87-b1f7b6764aee","Type":"ContainerDied","Data":"ecd0be992badf00e36d1ac55ad310ed7914a069bf6f52ab7c0a53e095e33e169"} Nov 01 20:37:45.078867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:45.078833 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:46.164497 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:46.164450 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:37:46.204426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.204393 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:46.251819 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.251778 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-xks-ca\") pod \"0a799591-35e2-4292-8f87-b1f7b6764aee\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " Nov 01 20:37:46.252003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.251891 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-openshift-ca\") pod \"0a799591-35e2-4292-8f87-b1f7b6764aee\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " Nov 01 20:37:46.252003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.251995 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a799591-35e2-4292-8f87-b1f7b6764aee-kube-api-access-g9vmm\" (UniqueName: \"kubernetes.io/projected/0a799591-35e2-4292-8f87-b1f7b6764aee-kube-api-access-g9vmm\") pod \"0a799591-35e2-4292-8f87-b1f7b6764aee\" (UID: \"0a799591-35e2-4292-8f87-b1f7b6764aee\") " Nov 01 20:37:46.252320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.252293 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-openshift-ca" pod "0a799591-35e2-4292-8f87-b1f7b6764aee" (UID: "0a799591-35e2-4292-8f87-b1f7b6764aee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:37:46.252504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.252484 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:46.254206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.254133 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-xks-ca" pod "0a799591-35e2-4292-8f87-b1f7b6764aee" (UID: "0a799591-35e2-4292-8f87-b1f7b6764aee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:37:46.254306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.254206 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a799591-35e2-4292-8f87-b1f7b6764aee-kube-api-access-g9vmm" pod "0a799591-35e2-4292-8f87-b1f7b6764aee" (UID: "0a799591-35e2-4292-8f87-b1f7b6764aee"). InnerVolumeSpecName "kube-api-access-g9vmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:46.353914 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:46.353826 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b8ed0bcb9390: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:46.353914 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:46.353859 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:46.353914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.353856 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vmm\" (UniqueName: \"kubernetes.io/projected/0a799591-35e2-4292-8f87-b1f7b6764aee-kube-api-access-g9vmm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:46.353914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.353888 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a799591-35e2-4292-8f87-b1f7b6764aee-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:46.354185 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:37:46.353923 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:38:18.353901819 +0000 UTC m=+2106.028161066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:37:46.750857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.750821 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl"] Nov 01 20:37:46.935860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:46.935825 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:47.054531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:47.054438 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" Nov 01 20:37:47.054531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:47.054452 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7" event={"ID":"0a799591-35e2-4292-8f87-b1f7b6764aee","Type":"ContainerDied","Data":"814e7e27f0f7d500e92a6d962bdc3d19d4c04807beafd716261e5f34bed0ab0a"} Nov 01 20:37:47.054531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:47.054501 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="814e7e27f0f7d500e92a6d962bdc3d19d4c04807beafd716261e5f34bed0ab0a" Nov 01 20:37:47.221005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:47.220967 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:37:47.719230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:47.719196 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:48.000585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:48.000493 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:48.948138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:48.948085 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:49.888182 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:49.888150 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:52.005387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:52.005283 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:37:52.005387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:52.005382 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:37:54.393148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.393110 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj"] Nov 01 20:37:54.394083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.394047 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0a799591-35e2-4292-8f87-b1f7b6764aee" containerName="revoke-keys" Nov 01 20:37:54.394083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.394072 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a799591-35e2-4292-8f87-b1f7b6764aee" containerName="revoke-keys" Nov 01 20:37:54.394373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.394219 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a799591-35e2-4292-8f87-b1f7b6764aee" containerName="revoke-keys" Nov 01 20:37:54.398314 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.398290 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.452280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.452242 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj"] Nov 01 20:37:54.478382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.471125 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tht5m\" (UniqueName: \"kubernetes.io/projected/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-kube-api-access-tht5m\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.478382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.471295 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.478382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.471356 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.495778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.495629 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:37:54.496458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.496431 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:37:54.496613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.496514 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:37:54.499383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.499357 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:37:54.573021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.572973 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.573284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.573265 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tht5m\" (UniqueName: \"kubernetes.io/projected/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-kube-api-access-tht5m\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.573695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.573669 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.574925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.574885 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.577161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.577129 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.603700 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.603652 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:54.630547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.630512 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tht5m\" (UniqueName: \"kubernetes.io/projected/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-kube-api-access-tht5m\") pod \"maas-api-revoke-sub-5bbde14f-xzwrj\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:54.716920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:54.716878 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:55.121311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:55.121282 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj"] Nov 01 20:37:55.286504 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:55.286462 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:55.774996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:55.774947 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:55.947073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:55.947024 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:37:56.126126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:56.126092 5633 generic.go:356] "Generic (PLEG): container finished" podID="d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" containerID="d2f0cd41760382faa5449ca79ae2e92797c58bf9133b099f217a645088a5e74c" exitCode=0 Nov 01 20:37:56.126312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:56.126144 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" event={"ID":"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74","Type":"ContainerDied","Data":"d2f0cd41760382faa5449ca79ae2e92797c58bf9133b099f217a645088a5e74c"} Nov 01 20:37:56.126312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:56.126192 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" event={"ID":"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74","Type":"ContainerStarted","Data":"0dd71233c9393d86a44a49568f24043ad9aeab9de442ec9476d4a68d8d80cb03"} Nov 01 20:37:57.240104 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.240063 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:37:57.588304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.588274 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:57.652419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.652377 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-xks-ca\") pod \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " Nov 01 20:37:57.652629 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.652573 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-openshift-ca\") pod \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " Nov 01 20:37:57.652703 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.652657 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-kube-api-access-tht5m\" (UniqueName: \"kubernetes.io/projected/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-kube-api-access-tht5m\") pod \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\" (UID: \"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74\") " Nov 01 20:37:57.653467 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.652963 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-openshift-ca" pod "d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" (UID: "d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:37:57.653693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.653670 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:57.654967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.654938 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-xks-ca" pod "d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" (UID: "d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:37:57.655283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.655260 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-kube-api-access-tht5m" pod "d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" (UID: "d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74"). InnerVolumeSpecName "kube-api-access-tht5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:57.755884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.755525 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:57.755884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:57.755561 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tht5m\" (UniqueName: \"kubernetes.io/projected/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74-kube-api-access-tht5m\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:58.198113 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:58.198080 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" Nov 01 20:37:58.198322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:58.198116 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj" event={"ID":"d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74","Type":"ContainerDied","Data":"0dd71233c9393d86a44a49568f24043ad9aeab9de442ec9476d4a68d8d80cb03"} Nov 01 20:37:58.198322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:58.198166 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd71233c9393d86a44a49568f24043ad9aeab9de442ec9476d4a68d8d80cb03" Nov 01 20:37:59.805243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:37:59.805204 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:00.835533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:00.835495 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:01.773248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:01.773196 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" containerID="cri-o://97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a" gracePeriod=45 Nov 01 20:38:02.063365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.063316 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:38:02.222498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.222460 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-dshm\") pod \"123292cc-88ae-449c-80a1-793b135d8a4a\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " Nov 01 20:38:02.222670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.222580 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-model-cache\") pod \"123292cc-88ae-449c-80a1-793b135d8a4a\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " Nov 01 20:38:02.222670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.222630 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123292cc-88ae-449c-80a1-793b135d8a4a-kube-api-access-mjrml\" (UniqueName: \"kubernetes.io/projected/123292cc-88ae-449c-80a1-793b135d8a4a-kube-api-access-mjrml\") pod \"123292cc-88ae-449c-80a1-793b135d8a4a\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " Nov 01 20:38:02.222796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.222780 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-tmp-dir\") pod \"123292cc-88ae-449c-80a1-793b135d8a4a\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " Nov 01 20:38:02.222872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.222858 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123292cc-88ae-449c-80a1-793b135d8a4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/123292cc-88ae-449c-80a1-793b135d8a4a-tls-certs\") pod \"123292cc-88ae-449c-80a1-793b135d8a4a\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " Nov 01 20:38:02.222936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.222892 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-model-cache" pod "123292cc-88ae-449c-80a1-793b135d8a4a" (UID: "123292cc-88ae-449c-80a1-793b135d8a4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.222936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.222906 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-home\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-home\") pod \"123292cc-88ae-449c-80a1-793b135d8a4a\" (UID: \"123292cc-88ae-449c-80a1-793b135d8a4a\") " Nov 01 20:38:02.223142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.223123 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-home" pod "123292cc-88ae-449c-80a1-793b135d8a4a" (UID: "123292cc-88ae-449c-80a1-793b135d8a4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.226489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.223141 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-tmp-dir" pod "123292cc-88ae-449c-80a1-793b135d8a4a" (UID: "123292cc-88ae-449c-80a1-793b135d8a4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.226489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.223670 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.226489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.223696 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.226489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.223710 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.226489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.225846 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123292cc-88ae-449c-80a1-793b135d8a4a-kube-api-access-mjrml" pod "123292cc-88ae-449c-80a1-793b135d8a4a" (UID: "123292cc-88ae-449c-80a1-793b135d8a4a"). InnerVolumeSpecName "kube-api-access-mjrml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:02.228230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.228198 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123292cc-88ae-449c-80a1-793b135d8a4a-tls-certs" pod "123292cc-88ae-449c-80a1-793b135d8a4a" (UID: "123292cc-88ae-449c-80a1-793b135d8a4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:38:02.229690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.229665 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-dshm" pod "123292cc-88ae-449c-80a1-793b135d8a4a" (UID: "123292cc-88ae-449c-80a1-793b135d8a4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.235724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.235693 5633 generic.go:356] "Generic (PLEG): container finished" podID="123292cc-88ae-449c-80a1-793b135d8a4a" containerID="97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a" exitCode=0 Nov 01 20:38:02.235877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.235740 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" Nov 01 20:38:02.235877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.235775 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" event={"ID":"123292cc-88ae-449c-80a1-793b135d8a4a","Type":"ContainerDied","Data":"97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a"} Nov 01 20:38:02.235877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.235825 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl" event={"ID":"123292cc-88ae-449c-80a1-793b135d8a4a","Type":"ContainerDied","Data":"f35615ab5a0af43cf94f64c67bbcade190df70a9330cc36e6778ab1b2dfd6947"} Nov 01 20:38:02.235877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.235839 5633 scope.go:122] "RemoveContainer" containerID="97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a" Nov 01 20:38:02.267724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.267698 5633 scope.go:122] "RemoveContainer" containerID="97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a" Nov 01 20:38:02.268162 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:02.268135 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a\": container with ID starting with 97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a not found: ID does not exist" containerID="97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a" Nov 01 20:38:02.268289 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.268167 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a"} err="failed to get container status \"97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a\": rpc error: code = NotFound desc = could not find container \"97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a\": container with ID starting with 97c60e3d5720301bc4b256bc828e59ff89da6b17d1a6e9ab8bc68d08fd50437a not found: ID does not exist" Nov 01 20:38:02.325878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.325771 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/123292cc-88ae-449c-80a1-793b135d8a4a-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.325878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.325805 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/123292cc-88ae-449c-80a1-793b135d8a4a-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.325878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.325819 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrml\" (UniqueName: \"kubernetes.io/projected/123292cc-88ae-449c-80a1-793b135d8a4a-kube-api-access-mjrml\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.400139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.400103 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl"] Nov 01 20:38:02.404362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.404315 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-2-w0-d204f4-kserve-69f7cb6bb6-hwxzl"] Nov 01 20:38:02.468526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:02.468486 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" path="/var/lib/kubelet/pods/123292cc-88ae-449c-80a1-793b135d8a4a/volumes" Nov 01 20:38:04.603856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:04.603819 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:05.198828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:05.198777 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:38:06.579508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:06.579472 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:07.220925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:07.220886 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:38:08.819827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.819795 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798"] Nov 01 20:38:08.820483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.820467 5633 cpu_manager.go:414] "containerMap: removing container" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" Nov 01 20:38:08.820551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.820484 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" Nov 01 20:38:08.820551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.820500 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" containerName="revoke-keys" Nov 01 20:38:08.820551 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.820507 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" containerName="revoke-keys" Nov 01 20:38:08.820660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.820638 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="123292cc-88ae-449c-80a1-793b135d8a4a" containerName="main" Nov 01 20:38:08.820660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.820647 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" containerName="revoke-keys" Nov 01 20:38:08.824374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.824320 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.875920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.875885 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\"" Nov 01 20:38:08.922600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.922551 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.922994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.922968 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.923181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.923163 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.923399 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.923380 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.923573 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.923559 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.923779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.923765 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.923879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.923866 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.923972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.923959 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82zw\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-kube-api-access-s82zw\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.924132 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.924118 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:08.952486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:08.952424 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798"] Nov 01 20:38:09.026115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026082 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026121 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026146 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82zw\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-kube-api-access-s82zw\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026214 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026292 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026553 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026596 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026642 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026706 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026723 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026793 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.026980 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026950 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.027069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.026959 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.027069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.027030 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.029181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.029160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.029300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.029278 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.083990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.083914 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.084179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.084133 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82zw\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-kube-api-access-s82zw\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.139405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.139369 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:09.173309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.173271 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:09.391784 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.391685 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:09.573034 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:38:09.573006 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3b044f_05e5_4d1d_bdf2_c7726bcaaa28.slice/crio-97e97161c3cac031155351b672d72748fe4629411deba6c36d850ecaef95b429 WatchSource:0}: Error finding container 97e97161c3cac031155351b672d72748fe4629411deba6c36d850ecaef95b429: Status 404 returned error can't find the container with id 97e97161c3cac031155351b672d72748fe4629411deba6c36d850ecaef95b429 Nov 01 20:38:09.573981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.573956 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798"] Nov 01 20:38:09.576059 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.576029 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:09.576146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.576092 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:09.576146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:09.576121 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:10.319554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:10.319515 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" event={"ID":"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28","Type":"ContainerStarted","Data":"9009ce5a3f1c745c314feac833e66faab09bf03162130d564ccb1c348dbf7105"} Nov 01 20:38:10.320009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:10.319586 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" event={"ID":"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28","Type":"ContainerStarted","Data":"97e97161c3cac031155351b672d72748fe4629411deba6c36d850ecaef95b429"} Nov 01 20:38:10.510401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:10.486817 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podStartSLOduration=2.486797973 podStartE2EDuration="2.486797973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:10.484556921 +0000 UTC m=+2098.158816177" watchObservedRunningTime="2026-11-01 20:38:10.486797973 +0000 UTC m=+2098.161057229" Nov 01 20:38:11.139871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:11.139833 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:11.141802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:11.141778 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 20:38:11.141900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:11.141829 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 20:38:12.140717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:12.140680 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 20:38:12.141150 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:12.140734 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 20:38:13.139827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:13.139789 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 20:38:13.140023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:13.139840 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 20:38:14.058106 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:14.058075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:14.139936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:14.139900 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 20:38:14.140128 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:14.139957 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 20:38:14.259407 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:14.259371 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:15.082587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:15.082538 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:15.139744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:15.139709 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 20:38:15.139933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:15.139760 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 20:38:16.139853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:16.139808 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 20:38:16.140448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:16.139896 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 20:38:16.159729 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:16.159690 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:38:17.220526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:17.220485 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:38:17.515062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:17.514975 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:17.516039 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:17.515906 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:17.516772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:17.516747 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:18.385063 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:18.385034 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b8ed0bcb9390: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:38:18.385063 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:18.385058 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58777c5b7f-wl887: secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:38:18.385610 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:18.385111 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials podName:37eff729-76d4-41a0-a88c-5cf03cce0537 nodeName:}" failed. No retries permitted until 2026-11-01 20:39:22.385095588 +0000 UTC m=+2170.059354820 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials") pod "payload-processing-external-model-58777c5b7f-wl887" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537") : secret "agc-cred-b8ed0bcb9390" not found Nov 01 20:38:18.997246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:18.997204 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:23.863356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:23.862550 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:24.079212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:24.079156 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:25.256874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:25.256824 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:27.220370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:27.220314 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:38:28.047392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:28.047355 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2"] Nov 01 20:38:28.096274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:28.096241 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3dece-t7mw2"] Nov 01 20:38:28.467739 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:28.467700 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a53af7e-5492-4d2a-96d2-830f9387e5a4" path="/var/lib/kubelet/pods/3a53af7e-5492-4d2a-96d2-830f9387e5a4/volumes" Nov 01 20:38:28.477242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:28.477208 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:29.469378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:29.469291 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:29.666668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:29.666632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:29.795493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:29.795381 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:31.110030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:31.109991 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:31.284239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:31.284203 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:32.658118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:32.658079 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:32.920475 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:32.920380 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:33.849466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:33.849419 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w"] Nov 01 20:38:33.850929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:33.850898 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:38:33.879166 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:33.879131 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9"] Nov 01 20:38:33.883558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:33.883534 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:33.935208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:33.935174 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-63b1876e63ce\"" Nov 01 20:38:33.935208 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:33.935200 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-401f7dd59863\"" Nov 01 20:38:33.964400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:33.964064 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9"] Nov 01 20:38:34.036443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.036389 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.036652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.036517 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zndw\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-kube-api-access-2zndw\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.036652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.036580 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-plugins-config-volume\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.036797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.036784 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-routing-overlay-volume\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.138613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.138517 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zndw\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-kube-api-access-2zndw\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.138613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.138591 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-plugins-config-volume\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.138818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.138751 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-routing-overlay-volume\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.138818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.138794 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.139437 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.139410 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-plugins-config-volume\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.139564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.139486 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-routing-overlay-volume\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.141811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.141788 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.215426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.215324 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zndw\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-kube-api-access-2zndw\") pod \"payload-processing-external-model-6dcf58fd8-8dvn9\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.363217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.363180 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:34.499521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.499477 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:34.879233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:34.879198 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9"] Nov 01 20:38:34.879917 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:38:34.879883 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e4a7e2_cfd5_4459_8714_edf33451e2ac.slice/crio-09fff9e0e2e1cf72f2dff9683d1e09fe55b6bb12583a5b6dd35dd0815bd01ae2 WatchSource:0}: Error finding container 09fff9e0e2e1cf72f2dff9683d1e09fe55b6bb12583a5b6dd35dd0815bd01ae2: Status 404 returned error can't find the container with id 09fff9e0e2e1cf72f2dff9683d1e09fe55b6bb12583a5b6dd35dd0815bd01ae2 Nov 01 20:38:35.572464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:35.572429 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" event={"ID":"47e4a7e2-cfd5-4459-8714-edf33451e2ac","Type":"ContainerStarted","Data":"641fd16bed56eb6c5854ac024dd3af2aa030868b57618598f55a6e24c839b66b"} Nov 01 20:38:35.572669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:35.572482 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" event={"ID":"47e4a7e2-cfd5-4459-8714-edf33451e2ac","Type":"ContainerStarted","Data":"09fff9e0e2e1cf72f2dff9683d1e09fe55b6bb12583a5b6dd35dd0815bd01ae2"} Nov 01 20:38:35.572669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:35.572573 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:35.685324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:35.685266 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" podStartSLOduration=2.685251375 podStartE2EDuration="2.685251375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:35.683974223 +0000 UTC m=+2123.358233478" watchObservedRunningTime="2026-11-01 20:38:35.685251375 +0000 UTC m=+2123.359510630" Nov 01 20:38:38.046503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:38.046472 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz"] Nov 01 20:38:38.099200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:38.099160 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96383c53-k8trz"] Nov 01 20:38:38.470064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:38.470028 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a812ddaf-b525-44b0-9f84-f421ee4f43f6" path="/var/lib/kubelet/pods/a812ddaf-b525-44b0-9f84-f421ee4f43f6/volumes" Nov 01 20:38:38.850810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:38.850694 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" containerID="cri-o://c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440" gracePeriod=25 Nov 01 20:38:39.004834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.004802 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:39.069490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.069454 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd"] Nov 01 20:38:39.075127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.075098 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.105774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.105741 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.105922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.105787 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.105922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.105827 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.106009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.105950 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.106009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.105995 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.106089 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.106060 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.106158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.106144 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.106200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.106168 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6705ac81-8c30-4af3-85c0-195703dc08e7-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.106244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.106221 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2bx\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-kube-api-access-mq2bx\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.142726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.142694 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:38:39.171731 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.171687 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd"] Nov 01 20:38:39.207258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207218 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-plugins-config-volume\") pod \"30293fac-754e-4991-98d4-3ca8397d81ad\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " Nov 01 20:38:39.207492 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207458 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-provider-credentials\") pod \"30293fac-754e-4991-98d4-3ca8397d81ad\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " Nov 01 20:38:39.207570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207511 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-kube-api-access-gg66b\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-kube-api-access-gg66b\") pod \"30293fac-754e-4991-98d4-3ca8397d81ad\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " Nov 01 20:38:39.207631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207588 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-routing-overlay-volume\") pod \"30293fac-754e-4991-98d4-3ca8397d81ad\" (UID: \"30293fac-754e-4991-98d4-3ca8397d81ad\") " Nov 01 20:38:39.207798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207765 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-plugins-config-volume" pod "30293fac-754e-4991-98d4-3ca8397d81ad" (UID: "30293fac-754e-4991-98d4-3ca8397d81ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:39.207798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207788 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.207971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207824 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6705ac81-8c30-4af3-85c0-195703dc08e7-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.207971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207899 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2bx\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-kube-api-access-mq2bx\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.207988 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208011 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-routing-overlay-volume" pod "30293fac-754e-4991-98d4-3ca8397d81ad" (UID: "30293fac-754e-4991-98d4-3ca8397d81ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:39.208087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208031 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208060 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208186 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208304 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208241 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208315 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208320 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.208575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208550 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:39.208575 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208570 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30293fac-754e-4991-98d4-3ca8397d81ad-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:39.208683 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208618 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.209146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.208901 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.209146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.209045 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6705ac81-8c30-4af3-85c0-195703dc08e7-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.209146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.209103 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.210496 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.210427 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-provider-credentials" pod "30293fac-754e-4991-98d4-3ca8397d81ad" (UID: "30293fac-754e-4991-98d4-3ca8397d81ad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:39.211986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.211903 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.211986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.211909 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-kube-api-access-gg66b" pod "30293fac-754e-4991-98d4-3ca8397d81ad" (UID: "30293fac-754e-4991-98d4-3ca8397d81ad"). InnerVolumeSpecName "kube-api-access-gg66b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:39.227360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.227264 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.267810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.267753 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.268138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.268118 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2bx\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-kube-api-access-mq2bx\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.310116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.310074 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:39.310116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.310104 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg66b\" (UniqueName: \"kubernetes.io/projected/30293fac-754e-4991-98d4-3ca8397d81ad-kube-api-access-gg66b\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:39.393690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.393606 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:39.504493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.504456 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:39.603156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.603116 5633 generic.go:356] "Generic (PLEG): container finished" podID="30293fac-754e-4991-98d4-3ca8397d81ad" containerID="c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440" exitCode=0 Nov 01 20:38:39.603366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.603167 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" Nov 01 20:38:39.603366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.603197 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" event={"ID":"30293fac-754e-4991-98d4-3ca8397d81ad","Type":"ContainerDied","Data":"c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440"} Nov 01 20:38:39.603366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.603253 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w" event={"ID":"30293fac-754e-4991-98d4-3ca8397d81ad","Type":"ContainerDied","Data":"493d715df428a886980e9759960acc532d48302e734b87e505cee239144466af"} Nov 01 20:38:39.603366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.603277 5633 scope.go:122] "RemoveContainer" containerID="c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440" Nov 01 20:38:39.614791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.614772 5633 scope.go:122] "RemoveContainer" containerID="c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440" Nov 01 20:38:39.615082 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:39.615059 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440\": container with ID starting with c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440 not found: ID does not exist" containerID="c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440" Nov 01 20:38:39.615172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.615092 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440"} err="failed to get container status \"c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440\": rpc error: code = NotFound desc = could not find container \"c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440\": container with ID starting with c9d9dab6e3291b0e606ae81f96ea7e4e39a0e7b7aa9571742cbb1ca36f7a2440 not found: ID does not exist" Nov 01 20:38:39.668639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.668541 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:39.770519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.770479 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w"] Nov 01 20:38:39.774580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.774557 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-hgf8w"] Nov 01 20:38:39.832714 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:38:39.832685 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6705ac81_8c30_4af3_85c0_195703dc08e7.slice/crio-85e88954b95a949d78c6d914183e4f3d9e9570caf612f6010bac425604622afb WatchSource:0}: Error finding container 85e88954b95a949d78c6d914183e4f3d9e9570caf612f6010bac425604622afb: Status 404 returned error can't find the container with id 85e88954b95a949d78c6d914183e4f3d9e9570caf612f6010bac425604622afb Nov 01 20:38:39.833689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.833660 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd"] Nov 01 20:38:39.835634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.835599 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:39.835751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.835672 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:39.835751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:39.835718 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:40.467993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:40.467957 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" path="/var/lib/kubelet/pods/30293fac-754e-4991-98d4-3ca8397d81ad/volumes" Nov 01 20:38:40.619478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:40.619436 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" event={"ID":"6705ac81-8c30-4af3-85c0-195703dc08e7","Type":"ContainerStarted","Data":"568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34"} Nov 01 20:38:40.619780 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:40.619746 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" event={"ID":"6705ac81-8c30-4af3-85c0-195703dc08e7","Type":"ContainerStarted","Data":"85e88954b95a949d78c6d914183e4f3d9e9570caf612f6010bac425604622afb"} Nov 01 20:38:40.836400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:40.836263 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" podStartSLOduration=1.8362446879999998 podStartE2EDuration="1.836244688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:40.7858097 +0000 UTC m=+2128.460068956" watchObservedRunningTime="2026-11-01 20:38:40.836244688 +0000 UTC m=+2128.510503943" Nov 01 20:38:41.018431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.018394 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:41.398357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.394794 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:41.411309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.410564 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 20:38:41.411309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.410616 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" podUID="6705ac81-8c30-4af3-85c0-195703dc08e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 20:38:41.428519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.427923 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:38:41.474444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.474402 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd"] Nov 01 20:38:41.481877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.481841 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798"] Nov 01 20:38:41.482126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.482094 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" containerID="cri-o://9009ce5a3f1c745c314feac833e66faab09bf03162130d564ccb1c348dbf7105" gracePeriod=30 Nov 01 20:38:41.532743 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.532706 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:41.628523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:41.628131 5633 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" secret="" err="secret \"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\" not found" Nov 01 20:38:42.399360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:42.397737 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 20:38:42.399360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:42.397796 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" podUID="6705ac81-8c30-4af3-85c0-195703dc08e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 20:38:42.698210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:42.698096 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" podUID="6705ac81-8c30-4af3-85c0-195703dc08e7" containerName="istio-proxy" containerID="cri-o://568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34" gracePeriod=30 Nov 01 20:38:44.089359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:44.088493 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:38:44.753361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:44.749821 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:38:44.757761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:44.757719 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:45.801938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:45.801860 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:38:45.803355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:45.803308 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:38:46.117655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:46.116654 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:38:46.711357 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:46.709495 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:38:46.740739 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:46.740556 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:38:46.813409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:46.806568 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:38:47.176424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.153808 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:38:47.176424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.154351 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:38:47.263357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.258865 5633 generic.go:356] "Generic (PLEG): container finished" podID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerID="9009ce5a3f1c745c314feac833e66faab09bf03162130d564ccb1c348dbf7105" exitCode=0 Nov 01 20:38:47.263357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.259160 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" event={"ID":"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28","Type":"ContainerDied","Data":"9009ce5a3f1c745c314feac833e66faab09bf03162130d564ccb1c348dbf7105"} Nov 01 20:38:47.436430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.435387 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58777c5b7f-wl887"] Nov 01 20:38:47.562122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.562087 5633 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 20:38:47.562451 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.562413 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 20:38:47.653096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:47.651724 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:38:48.178487 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.178426 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 01 20:38:48.458418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.452240 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:48.566360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.562391 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-credential-socket\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.566360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.562476 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-podinfo\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.566360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.562544 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-certs\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.566360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.562604 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-token\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-token\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.566360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.562666 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-kube-api-access-s82zw\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-kube-api-access-s82zw\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.566360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.562762 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-socket\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.596020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.586760 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-socket" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:48.596020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.587113 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-credential-socket" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:48.621052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.621008 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-envoy\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.641586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.623440 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istiod-ca-cert\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.642156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.642125 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-data\") pod \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\" (UID: \"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28\") " Nov 01 20:38:48.667537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.635971 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-certs" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:48.667537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.651138 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istiod-ca-cert" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:48.691358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.686176 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-podinfo" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:38:48.750416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.732964 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-data" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:48.887356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.860273 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-envoy" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:48.903358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.878989 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-kube-api-access-s82zw" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "kube-api-access-s82zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:48.903358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.879426 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-token" pod "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" (UID: "2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:48.903358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.877939 5633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:38:48.925371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.923815 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" Nov 01 20:38:48.945282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.944057 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.945282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.944483 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82zw\" (UniqueName: \"kubernetes.io/projected/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-kube-api-access-s82zw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.945282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.945223 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.994489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.967622 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.994489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.968043 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.994489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.968455 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.994489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.968485 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.994489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.968785 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:48.994489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:48.968809 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:49.058640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:49.058582 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 01 20:38:49.067375 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:49.061022 5633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3b044f_05e5_4d1d_bdf2_c7726bcaaa28.slice/crio-9009ce5a3f1c745c314feac833e66faab09bf03162130d564ccb1c348dbf7105.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3b044f_05e5_4d1d_bdf2_c7726bcaaa28.slice/crio-conmon-9009ce5a3f1c745c314feac833e66faab09bf03162130d564ccb1c348dbf7105.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:38:49.115421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:49.100108 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798" event={"ID":"2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28","Type":"ContainerDied","Data":"97e97161c3cac031155351b672d72748fe4629411deba6c36d850ecaef95b429"} Nov 01 20:38:49.115421 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:49.100188 5633 scope.go:122] "RemoveContainer" containerID="9009ce5a3f1c745c314feac833e66faab09bf03162130d564ccb1c348dbf7105" Nov 01 20:38:49.181896 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:49.181850 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:38:49.503405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:49.495104 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798"] Nov 01 20:38:49.574413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:49.571567 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-fs798"] Nov 01 20:38:49.782914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:49.782888 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:38:50.086615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:50.086552 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl"] Nov 01 20:38:50.143716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:50.127783 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d198ed-bxrpl"] Nov 01 20:38:50.518833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:50.517638 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" path="/var/lib/kubelet/pods/2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28/volumes" Nov 01 20:38:50.518833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:50.518390 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650a9d50-e5ca-4c52-998b-1df06b5c139e" path="/var/lib/kubelet/pods/650a9d50-e5ca-4c52-998b-1df06b5c139e/volumes" Nov 01 20:38:50.629466 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:50.629292 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:52.437225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.437165 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" podUID="37eff729-76d4-41a0-a88c-5cf03cce0537" containerName="payload-processing" containerID="cri-o://08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab" gracePeriod=25 Nov 01 20:38:52.768243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.768213 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:38:52.861894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.861860 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-routing-overlay-volume\") pod \"37eff729-76d4-41a0-a88c-5cf03cce0537\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " Nov 01 20:38:52.862096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.862065 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials\") pod \"37eff729-76d4-41a0-a88c-5cf03cce0537\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " Nov 01 20:38:52.862164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.862102 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-plugins-config-volume\") pod \"37eff729-76d4-41a0-a88c-5cf03cce0537\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " Nov 01 20:38:52.862228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.862214 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-kube-api-access-jn9h5\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-kube-api-access-jn9h5\") pod \"37eff729-76d4-41a0-a88c-5cf03cce0537\" (UID: \"37eff729-76d4-41a0-a88c-5cf03cce0537\") " Nov 01 20:38:52.862284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.862233 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-routing-overlay-volume" pod "37eff729-76d4-41a0-a88c-5cf03cce0537" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:52.862713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.862599 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-plugins-config-volume" pod "37eff729-76d4-41a0-a88c-5cf03cce0537" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:52.862900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.862869 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:52.862900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.862888 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37eff729-76d4-41a0-a88c-5cf03cce0537-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:52.865113 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.865090 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-kube-api-access-jn9h5" pod "37eff729-76d4-41a0-a88c-5cf03cce0537" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537"). InnerVolumeSpecName "kube-api-access-jn9h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:52.865411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.865387 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials" pod "37eff729-76d4-41a0-a88c-5cf03cce0537" (UID: "37eff729-76d4-41a0-a88c-5cf03cce0537"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:52.964693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.964649 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:52.964693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:52.964696 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9h5\" (UniqueName: \"kubernetes.io/projected/37eff729-76d4-41a0-a88c-5cf03cce0537-kube-api-access-jn9h5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.098564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.098475 5633 generic.go:356] "Generic (PLEG): container finished" podID="37eff729-76d4-41a0-a88c-5cf03cce0537" containerID="08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab" exitCode=0 Nov 01 20:38:53.098564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.098531 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" Nov 01 20:38:53.098800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.098576 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" event={"ID":"37eff729-76d4-41a0-a88c-5cf03cce0537","Type":"ContainerDied","Data":"08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab"} Nov 01 20:38:53.098800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.098648 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58777c5b7f-wl887" event={"ID":"37eff729-76d4-41a0-a88c-5cf03cce0537","Type":"ContainerDied","Data":"7ba2faaebee4367e27f66ffcb0c135726459aa84418ade4de5baa6778e70f13a"} Nov 01 20:38:53.098800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.098685 5633 scope.go:122] "RemoveContainer" containerID="08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab" Nov 01 20:38:53.110752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.110721 5633 scope.go:122] "RemoveContainer" containerID="08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab" Nov 01 20:38:53.111084 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:53.111065 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab\": container with ID starting with 08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab not found: ID does not exist" containerID="08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab" Nov 01 20:38:53.111164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.111092 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab"} err="failed to get container status \"08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab\": rpc error: code = NotFound desc = could not find container \"08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab\": container with ID starting with 08180a167d34f996d27d1caddaa526cfa3312276240d1d4b15e8b6f439425fab not found: ID does not exist" Nov 01 20:38:53.266107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.266071 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58777c5b7f-wl887"] Nov 01 20:38:53.272915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:53.272886 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58777c5b7f-wl887"] Nov 01 20:38:54.393794 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:54.393762 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:54.479387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:54.479322 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37eff729-76d4-41a0-a88c-5cf03cce0537" path="/var/lib/kubelet/pods/37eff729-76d4-41a0-a88c-5cf03cce0537/volumes" Nov 01 20:38:55.118146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:55.118056 5633 generic.go:356] "Generic (PLEG): container finished" podID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerID="fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7" exitCode=0 Nov 01 20:38:55.118388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:55.118140 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" event={"ID":"76e7055e-4677-478f-bb8e-4423b79d72c0","Type":"ContainerDied","Data":"fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7"} Nov 01 20:38:55.118798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:55.118780 5633 scope.go:122] "RemoveContainer" containerID="fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7" Nov 01 20:38:55.120622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:55.120594 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:55.120708 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:55.120670 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:55.120751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:55.120706 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 20:38:55.312720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:55.312676 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:38:56.132984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.131563 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" event={"ID":"76e7055e-4677-478f-bb8e-4423b79d72c0","Type":"ContainerStarted","Data":"6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6"} Nov 01 20:38:56.428348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.428305 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:56.527311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527265 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-data\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527307 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-socket\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527527 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-podinfo\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527565 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6705ac81-8c30-4af3-85c0-195703dc08e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6705ac81-8c30-4af3-85c0-195703dc08e7-istiod-ca-cert\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527585 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-socket" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:56.527613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527596 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-data" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:56.527844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527606 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-kube-api-access-mq2bx\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-kube-api-access-mq2bx\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527670 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-credential-socket\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527695 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-certs\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527744 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-token\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-token\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.527844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527786 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-envoy\") pod \"6705ac81-8c30-4af3-85c0-195703dc08e7\" (UID: \"6705ac81-8c30-4af3-85c0-195703dc08e7\") " Nov 01 20:38:56.528123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527936 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-certs" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:56.528123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527968 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-credential-socket" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:56.528123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.527973 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6705ac81-8c30-4af3-85c0-195703dc08e7-istiod-ca-cert" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:56.528801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.528699 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.528801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.528726 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.528801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.528745 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6705ac81-8c30-4af3-85c0-195703dc08e7-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.528801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.528761 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.528801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.528777 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.530595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.530572 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-podinfo" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:38:56.530717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.530650 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-kube-api-access-mq2bx" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "kube-api-access-mq2bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:56.530815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.530799 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-envoy" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:56.531029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.531013 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-token" pod "6705ac81-8c30-4af3-85c0-195703dc08e7" (UID: "6705ac81-8c30-4af3-85c0-195703dc08e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:56.630738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.630698 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.630738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.630732 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.630738 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.630745 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6705ac81-8c30-4af3-85c0-195703dc08e7-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.630994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.630758 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq2bx\" (UniqueName: \"kubernetes.io/projected/6705ac81-8c30-4af3-85c0-195703dc08e7-kube-api-access-mq2bx\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:56.794355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.793735 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:38:56.794355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.793875 5633 patch_prober.go:29] interesting pod/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 01 20:38:56.794355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:56.793962 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 01 20:38:57.144450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.143745 5633 generic.go:356] "Generic (PLEG): container finished" podID="6705ac81-8c30-4af3-85c0-195703dc08e7" containerID="568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34" exitCode=0 Nov 01 20:38:57.144891 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.144716 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" Nov 01 20:38:57.146694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.146667 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" event={"ID":"6705ac81-8c30-4af3-85c0-195703dc08e7","Type":"ContainerDied","Data":"568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34"} Nov 01 20:38:57.146752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.146731 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd" event={"ID":"6705ac81-8c30-4af3-85c0-195703dc08e7","Type":"ContainerDied","Data":"85e88954b95a949d78c6d914183e4f3d9e9570caf612f6010bac425604622afb"} Nov 01 20:38:57.146798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.146764 5633 scope.go:122] "RemoveContainer" containerID="568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34" Nov 01 20:38:57.164115 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.164085 5633 scope.go:122] "RemoveContainer" containerID="568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34" Nov 01 20:38:57.164612 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:38:57.164587 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34\": container with ID starting with 568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34 not found: ID does not exist" containerID="568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34" Nov 01 20:38:57.164714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.164624 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34"} err="failed to get container status \"568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34\": rpc error: code = NotFound desc = could not find container \"568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34\": container with ID starting with 568f3a4a2c51bb00fff622981b59ee3ed82eccdb3c6b7e18f509c88b33e24d34 not found: ID does not exist" Nov 01 20:38:57.317426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.317392 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd"] Nov 01 20:38:57.364240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.364208 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-c5rdd"] Nov 01 20:38:57.806445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.806408 5633 patch_prober.go:29] interesting pod/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 01 20:38:57.807550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:57.807518 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 01 20:38:58.471121 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:58.471072 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6705ac81-8c30-4af3-85c0-195703dc08e7" path="/var/lib/kubelet/pods/6705ac81-8c30-4af3-85c0-195703dc08e7/volumes" Nov 01 20:38:58.801416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:58.796253 5633 patch_prober.go:29] interesting pod/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 01 20:38:58.801416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:58.796385 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 01 20:38:59.805536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:59.805490 5633 patch_prober.go:29] interesting pod/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 01 20:38:59.806266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:38:59.806230 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 01 20:39:00.044063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:00.044015 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk"] Nov 01 20:39:00.114859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:00.114767 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2da8f3-xkfwk"] Nov 01 20:39:00.324206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:00.324163 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:00.468234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:00.468201 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d3128f-9731-4ec4-8045-36461283f7b9" path="/var/lib/kubelet/pods/d9d3128f-9731-4ec4-8045-36461283f7b9/volumes" Nov 01 20:39:00.802858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:00.802755 5633 patch_prober.go:29] interesting pod/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 01 20:39:00.803158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:00.803124 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 01 20:39:01.929681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:01.929647 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:39:01.930218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:01.930125 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:39:01.933603 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:01.933576 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:39:02.100805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:02.100771 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs"] Nov 01 20:39:02.116916 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:02.116881 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613cb867-2wczs"] Nov 01 20:39:02.469298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:02.469263 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0407010c-1b6e-4166-9854-a7502c229b34" path="/var/lib/kubelet/pods/0407010c-1b6e-4166-9854-a7502c229b34/volumes" Nov 01 20:39:02.482457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:02.482428 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:05.367086 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:05.367047 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:06.710855 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:06.710820 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:10.075164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:10.075126 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:12.604897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:12.604862 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:13.329383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:13.329323 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f"] Nov 01 20:39:13.490806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:13.490768 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9"] Nov 01 20:39:13.691253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:13.691162 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:13.816380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:13.816346 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:15.145950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:15.145907 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:16.157842 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:16.157806 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:39:17.293960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:17.293920 5633 scope.go:122] "RemoveContainer" containerID="f8b3fd590a9d69ae0abb7c2ee36c05364599880370dff4951d6bb89cff652519" Nov 01 20:39:17.306116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:17.306091 5633 scope.go:122] "RemoveContainer" containerID="06bb8d44dc46507af276eabde62c02794ce6e96e3ca77f7175bb87b69f30f29d" Nov 01 20:39:18.048441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.048396 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m"] Nov 01 20:39:18.097672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.097637 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ddc240-sdb7m"] Nov 01 20:39:18.330194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.330106 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" podUID="39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" containerName="payload-processing" containerID="cri-o://8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b" gracePeriod=25 Nov 01 20:39:18.478300 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.478261 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9daa6ec7-64fa-4afc-b307-2f7e627a6ce8" path="/var/lib/kubelet/pods/9daa6ec7-64fa-4afc-b307-2f7e627a6ce8/volumes" Nov 01 20:39:18.491753 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.491702 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" podUID="f92ba181-03a2-43ca-8830-3f28285dd7ca" containerName="payload-pre-processing" containerID="cri-o://40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd" gracePeriod=25 Nov 01 20:39:18.539083 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.539046 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:18.624498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.624469 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:39:18.693530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.693481 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-plugins-config-volume\") pod \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " Nov 01 20:39:18.693770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.693750 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-kube-api-access-scc8x\" (UniqueName: \"kubernetes.io/projected/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-kube-api-access-scc8x\") pod \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\" (UID: \"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1\") " Nov 01 20:39:18.694044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.693869 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-plugins-config-volume" pod "39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" (UID: "39e8cafb-cbb3-4c40-964a-09a4b1ee72b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:18.694250 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.694231 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:18.697378 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.696646 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-kube-api-access-scc8x" pod "39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" (UID: "39e8cafb-cbb3-4c40-964a-09a4b1ee72b1"). InnerVolumeSpecName "kube-api-access-scc8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:18.759829 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.759798 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:39:18.795696 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.795660 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scc8x\" (UniqueName: \"kubernetes.io/projected/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1-kube-api-access-scc8x\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:18.896669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.896584 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92ba181-03a2-43ca-8830-3f28285dd7ca-kube-api-access-4d956\" (UniqueName: \"kubernetes.io/projected/f92ba181-03a2-43ca-8830-3f28285dd7ca-kube-api-access-4d956\") pod \"f92ba181-03a2-43ca-8830-3f28285dd7ca\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " Nov 01 20:39:18.896825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.896746 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f92ba181-03a2-43ca-8830-3f28285dd7ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92ba181-03a2-43ca-8830-3f28285dd7ca-plugins-config-volume\") pod \"f92ba181-03a2-43ca-8830-3f28285dd7ca\" (UID: \"f92ba181-03a2-43ca-8830-3f28285dd7ca\") " Nov 01 20:39:18.897206 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.897174 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92ba181-03a2-43ca-8830-3f28285dd7ca-plugins-config-volume" pod "f92ba181-03a2-43ca-8830-3f28285dd7ca" (UID: "f92ba181-03a2-43ca-8830-3f28285dd7ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:18.897503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.897478 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92ba181-03a2-43ca-8830-3f28285dd7ca-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:18.899184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.899152 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92ba181-03a2-43ca-8830-3f28285dd7ca-kube-api-access-4d956" pod "f92ba181-03a2-43ca-8830-3f28285dd7ca" (UID: "f92ba181-03a2-43ca-8830-3f28285dd7ca"). InnerVolumeSpecName "kube-api-access-4d956". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:18.999170 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:18.999136 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d956\" (UniqueName: \"kubernetes.io/projected/f92ba181-03a2-43ca-8830-3f28285dd7ca-kube-api-access-4d956\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:19.327827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.326893 5633 generic.go:356] "Generic (PLEG): container finished" podID="f92ba181-03a2-43ca-8830-3f28285dd7ca" containerID="40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd" exitCode=0 Nov 01 20:39:19.327827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.327122 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" Nov 01 20:39:19.327827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.327389 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" event={"ID":"f92ba181-03a2-43ca-8830-3f28285dd7ca","Type":"ContainerDied","Data":"40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd"} Nov 01 20:39:19.327827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.327440 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9" event={"ID":"f92ba181-03a2-43ca-8830-3f28285dd7ca","Type":"ContainerDied","Data":"98014ef59bd250f7c3fdfa67f480c1d0541c7741d1a74913aff89963ef7d74f1"} Nov 01 20:39:19.327827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.327469 5633 scope.go:122] "RemoveContainer" containerID="40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd" Nov 01 20:39:19.331000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.330971 5633 generic.go:356] "Generic (PLEG): container finished" podID="39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" containerID="8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b" exitCode=0 Nov 01 20:39:19.331476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.331096 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" Nov 01 20:39:19.331476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.331174 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" event={"ID":"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1","Type":"ContainerDied","Data":"8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b"} Nov 01 20:39:19.331476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.331220 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f" event={"ID":"39e8cafb-cbb3-4c40-964a-09a4b1ee72b1","Type":"ContainerDied","Data":"e8546cce1db73451c134f27d7e82f22ffb051265841b48141b1546621d33ebc3"} Nov 01 20:39:19.340970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.340915 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:19.348365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.348163 5633 scope.go:122] "RemoveContainer" containerID="40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd" Nov 01 20:39:19.348629 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:19.348605 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd\": container with ID starting with 40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd not found: ID does not exist" containerID="40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd" Nov 01 20:39:19.348723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.348639 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd"} err="failed to get container status \"40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd\": rpc error: code = NotFound desc = could not find container \"40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd\": container with ID starting with 40bfbb370df973b5a5babfbf7054bac80b756fd81cd3f2e4706d2b8a3767eccd not found: ID does not exist" Nov 01 20:39:19.348723 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.348658 5633 scope.go:122] "RemoveContainer" containerID="8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b" Nov 01 20:39:19.379550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.370353 5633 scope.go:122] "RemoveContainer" containerID="8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b" Nov 01 20:39:19.379550 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:19.370767 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b\": container with ID starting with 8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b not found: ID does not exist" containerID="8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b" Nov 01 20:39:19.379550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.370799 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b"} err="failed to get container status \"8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b\": rpc error: code = NotFound desc = could not find container \"8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b\": container with ID starting with 8efd50ff0183fc59b1343105d635610888b30f2bef208a972e0637742184fb4b not found: ID does not exist" Nov 01 20:39:19.500653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.500602 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9"] Nov 01 20:39:19.549433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.549395 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-012e1c-6847465b47-5pdt9"] Nov 01 20:39:19.669280 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.668283 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f"] Nov 01 20:39:19.676978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:19.672264 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-012e1c-dd7d747b7-r7p4f"] Nov 01 20:39:20.467894 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:20.467862 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" path="/var/lib/kubelet/pods/39e8cafb-cbb3-4c40-964a-09a4b1ee72b1/volumes" Nov 01 20:39:20.468301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:20.468288 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f92ba181-03a2-43ca-8830-3f28285dd7ca" path="/var/lib/kubelet/pods/f92ba181-03a2-43ca-8830-3f28285dd7ca/volumes" Nov 01 20:39:21.091364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:21.091245 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:24.312583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:24.312537 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:24.439077 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:24.439031 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:25.888704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:25.888645 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:26.013072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:26.013036 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:26.359156 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:26.359121 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:26.371361 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:26.371298 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr"] Nov 01 20:39:26.371585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:26.371563 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" containerID="cri-o://05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2" gracePeriod=30 Nov 01 20:39:27.190174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:27.190128 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:28.911924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:28.911867 5633 patch_prober.go:29] interesting pod/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:39:28.912469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:28.911943 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:39:29.839249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:29.839217 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:31.085935 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.085883 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:31.641948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.641885 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:39:31.702764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.702728 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-credential-socket\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.702949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.702850 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-envoy\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.702949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.702888 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-data\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.702949 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.702935 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-token\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-token\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.703107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.702965 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-socket\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.703107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.702992 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istiod-ca-cert\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.703107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703026 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-credential-socket" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:31.703226 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703205 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-certs\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.703226 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703207 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-data" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:31.703364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703267 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-socket" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:31.703364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703305 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-kube-api-access-8p55z\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-kube-api-access-8p55z\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.703488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703397 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-podinfo\") pod \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\" (UID: \"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca\") " Nov 01 20:39:31.703488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703417 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-certs" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:31.703488 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.703456 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istiod-ca-cert" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:31.704232 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.704205 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.704391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.704236 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.704391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.704251 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.704391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.704265 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.704391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.704279 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.705495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.705468 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-token" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:31.705495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.705481 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-envoy" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:31.705639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.705572 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-kube-api-access-8p55z" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "kube-api-access-8p55z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:31.705639 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.705617 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-podinfo" pod "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" (UID: "841cb4b8-f821-4cf4-9aa7-2b4ad71106ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:31.805309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.805277 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p55z\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-kube-api-access-8p55z\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.805309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.805307 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.805309 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.805317 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.805579 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.805343 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:31.808896 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:31.808869 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:39:32.438602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.438562 5633 generic.go:356] "Generic (PLEG): container finished" podID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerID="05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2" exitCode=0 Nov 01 20:39:32.439108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.438618 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" Nov 01 20:39:32.439108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.438640 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" event={"ID":"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca","Type":"ContainerDied","Data":"05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2"} Nov 01 20:39:32.439108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.438702 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr" event={"ID":"841cb4b8-f821-4cf4-9aa7-2b4ad71106ca","Type":"ContainerDied","Data":"d865e26cfc51e6e6fe2bd61f353978d8ebf480154c593e14ce2b261a4d1b79eb"} Nov 01 20:39:32.439108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.438732 5633 scope.go:122] "RemoveContainer" containerID="05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2" Nov 01 20:39:32.444144 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.444115 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv"] Nov 01 20:39:32.451577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.451558 5633 scope.go:122] "RemoveContainer" containerID="ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5" Nov 01 20:39:32.462457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.462423 5633 scope.go:122] "RemoveContainer" containerID="05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2" Nov 01 20:39:32.462770 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:32.462749 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2\": container with ID starting with 05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2 not found: ID does not exist" containerID="05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2" Nov 01 20:39:32.462835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.462781 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2"} err="failed to get container status \"05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2\": rpc error: code = NotFound desc = could not find container \"05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2\": container with ID starting with 05c94b6ea5279f4069fdae10e34ed26513172a67ae3c3a49bd55648c59aaf4c2 not found: ID does not exist" Nov 01 20:39:32.462835 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.462799 5633 scope.go:122] "RemoveContainer" containerID="ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5" Nov 01 20:39:32.463086 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:32.463070 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5\": container with ID starting with ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5 not found: ID does not exist" containerID="ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5" Nov 01 20:39:32.463141 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.463088 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5"} err="failed to get container status \"ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5\": rpc error: code = NotFound desc = could not find container \"ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5\": container with ID starting with ba8c46615961ca875593b2d6e363ec68742e66e139afed32571684cebca70cf5 not found: ID does not exist" Nov 01 20:39:32.548174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.548139 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr"] Nov 01 20:39:32.643970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.643937 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr"] Nov 01 20:39:32.658661 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.658581 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-012e1c-openshift-default-78786cfc65-95whr"] Nov 01 20:39:32.788154 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.788120 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:32.924274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:32.924243 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:33.680877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:33.680835 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:34.576073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:34.576029 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" path="/var/lib/kubelet/pods/841cb4b8-f821-4cf4-9aa7-2b4ad71106ca/volumes" Nov 01 20:39:35.840618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:35.839447 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:39:36.557464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:36.557430 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:37.192359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.191723 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m"] Nov 01 20:39:37.192359 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.191982 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" containerID="cri-o://0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399" gracePeriod=30 Nov 01 20:39:37.445419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.445314 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" podUID="20615658-7a77-4d68-b8e6-5720a7a1f6aa" containerName="payload-processing" containerID="cri-o://8cead925db64d72eaabb20bf14a9aeb90d593ef75e2e37efdb36436284ce5fb5" gracePeriod=25 Nov 01 20:39:37.548808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.548756 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" podUID="d52b3ede-52ee-460a-a7c1-c86e0389f4d4" containerName="payload-pre-processing" containerID="cri-o://d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22" gracePeriod=25 Nov 01 20:39:37.585380 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.585310 5633 generic.go:356] "Generic (PLEG): container finished" podID="20615658-7a77-4d68-b8e6-5720a7a1f6aa" containerID="8cead925db64d72eaabb20bf14a9aeb90d593ef75e2e37efdb36436284ce5fb5" exitCode=0 Nov 01 20:39:37.585554 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.585371 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" event={"ID":"20615658-7a77-4d68-b8e6-5720a7a1f6aa","Type":"ContainerDied","Data":"8cead925db64d72eaabb20bf14a9aeb90d593ef75e2e37efdb36436284ce5fb5"} Nov 01 20:39:37.725282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.725252 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:39:37.866230 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.866198 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:39:37.879489 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.879457 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20615658-7a77-4d68-b8e6-5720a7a1f6aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20615658-7a77-4d68-b8e6-5720a7a1f6aa-plugins-config-volume\") pod \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " Nov 01 20:39:37.879760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.879744 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20615658-7a77-4d68-b8e6-5720a7a1f6aa-kube-api-access-v875s\" (UniqueName: \"kubernetes.io/projected/20615658-7a77-4d68-b8e6-5720a7a1f6aa-kube-api-access-v875s\") pod \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\" (UID: \"20615658-7a77-4d68-b8e6-5720a7a1f6aa\") " Nov 01 20:39:37.879948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.879916 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20615658-7a77-4d68-b8e6-5720a7a1f6aa-plugins-config-volume" pod "20615658-7a77-4d68-b8e6-5720a7a1f6aa" (UID: "20615658-7a77-4d68-b8e6-5720a7a1f6aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:37.880771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.880314 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20615658-7a77-4d68-b8e6-5720a7a1f6aa-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:37.882173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.882141 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20615658-7a77-4d68-b8e6-5720a7a1f6aa-kube-api-access-v875s" pod "20615658-7a77-4d68-b8e6-5720a7a1f6aa" (UID: "20615658-7a77-4d68-b8e6-5720a7a1f6aa"). InnerVolumeSpecName "kube-api-access-v875s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:37.981449 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.981418 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-plugins-config-volume\") pod \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " Nov 01 20:39:37.981641 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.981463 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-kube-api-access-2wdmt\" (UniqueName: \"kubernetes.io/projected/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-kube-api-access-2wdmt\") pod \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\" (UID: \"d52b3ede-52ee-460a-a7c1-c86e0389f4d4\") " Nov 01 20:39:37.981827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.981800 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-plugins-config-volume" pod "d52b3ede-52ee-460a-a7c1-c86e0389f4d4" (UID: "d52b3ede-52ee-460a-a7c1-c86e0389f4d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:37.981871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.981834 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v875s\" (UniqueName: \"kubernetes.io/projected/20615658-7a77-4d68-b8e6-5720a7a1f6aa-kube-api-access-v875s\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:37.983755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:37.983733 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-kube-api-access-2wdmt" pod "d52b3ede-52ee-460a-a7c1-c86e0389f4d4" (UID: "d52b3ede-52ee-460a-a7c1-c86e0389f4d4"). InnerVolumeSpecName "kube-api-access-2wdmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:38.082818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.082780 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:38.082818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.082815 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wdmt\" (UniqueName: \"kubernetes.io/projected/d52b3ede-52ee-460a-a7c1-c86e0389f4d4-kube-api-access-2wdmt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:38.594392 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.594364 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" Nov 01 20:39:38.594902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.594364 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv" event={"ID":"20615658-7a77-4d68-b8e6-5720a7a1f6aa","Type":"ContainerDied","Data":"d8650bc544701ce5aaf3689cecdf51d0b059a5ff17ff8cde1d54dcfc6c5b5dc1"} Nov 01 20:39:38.594902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.594513 5633 scope.go:122] "RemoveContainer" containerID="8cead925db64d72eaabb20bf14a9aeb90d593ef75e2e37efdb36436284ce5fb5" Nov 01 20:39:38.595985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.595961 5633 generic.go:356] "Generic (PLEG): container finished" podID="d52b3ede-52ee-460a-a7c1-c86e0389f4d4" containerID="d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22" exitCode=0 Nov 01 20:39:38.596088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.596013 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" Nov 01 20:39:38.596088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.596060 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" event={"ID":"d52b3ede-52ee-460a-a7c1-c86e0389f4d4","Type":"ContainerDied","Data":"d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22"} Nov 01 20:39:38.596197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.596101 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr" event={"ID":"d52b3ede-52ee-460a-a7c1-c86e0389f4d4","Type":"ContainerDied","Data":"dea33223d6a5fb41ab0a1b74de9a59b7ba8b58db307d57f7fcfc70224b3010bd"} Nov 01 20:39:38.605808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.605789 5633 scope.go:122] "RemoveContainer" containerID="d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22" Nov 01 20:39:38.615543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.615523 5633 scope.go:122] "RemoveContainer" containerID="d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22" Nov 01 20:39:38.615831 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:38.615815 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22\": container with ID starting with d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22 not found: ID does not exist" containerID="d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22" Nov 01 20:39:38.615884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.615840 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22"} err="failed to get container status \"d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22\": rpc error: code = NotFound desc = could not find container \"d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22\": container with ID starting with d6b4c5eb7f2781166a21dc99bde613dd004f8a93e9f4397879c9d912485cce22 not found: ID does not exist" Nov 01 20:39:38.801126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.801046 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr"] Nov 01 20:39:38.804607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.804582 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13cbc9-6db598ff8c-nzzqr"] Nov 01 20:39:38.910441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.910406 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv"] Nov 01 20:39:38.914009 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:38.913986 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13cbc9-76579c5cc5-cjqqv"] Nov 01 20:39:39.109887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:39.109803 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:39.642112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:39.642075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:40.300550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.300513 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg"] Nov 01 20:39:40.301094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301080 5633 cpu_manager.go:414] "containerMap: removing container" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" Nov 01 20:39:40.301163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301095 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" Nov 01 20:39:40.301163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301116 5633 cpu_manager.go:414] "containerMap: removing container" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" Nov 01 20:39:40.301163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301123 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" Nov 01 20:39:40.301163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301132 5633 cpu_manager.go:414] "containerMap: removing container" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" Nov 01 20:39:40.301163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301141 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" Nov 01 20:39:40.301163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301159 5633 cpu_manager.go:414] "containerMap: removing container" podUID="20615658-7a77-4d68-b8e6-5720a7a1f6aa" containerName="payload-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301166 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20615658-7a77-4d68-b8e6-5720a7a1f6aa" containerName="payload-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301192 5633 cpu_manager.go:414] "containerMap: removing container" podUID="37eff729-76d4-41a0-a88c-5cf03cce0537" containerName="payload-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301200 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eff729-76d4-41a0-a88c-5cf03cce0537" containerName="payload-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301213 5633 cpu_manager.go:414] "containerMap: removing container" podUID="6705ac81-8c30-4af3-85c0-195703dc08e7" containerName="istio-proxy" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301219 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6705ac81-8c30-4af3-85c0-195703dc08e7" containerName="istio-proxy" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301240 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d52b3ede-52ee-460a-a7c1-c86e0389f4d4" containerName="payload-pre-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301247 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52b3ede-52ee-460a-a7c1-c86e0389f4d4" containerName="payload-pre-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301259 5633 cpu_manager.go:414] "containerMap: removing container" podUID="39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" containerName="payload-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301267 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" containerName="payload-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301295 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f92ba181-03a2-43ca-8830-3f28285dd7ca" containerName="payload-pre-processing" Nov 01 20:39:40.301425 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301302 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92ba181-03a2-43ca-8830-3f28285dd7ca" containerName="payload-pre-processing" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301467 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301481 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="30293fac-754e-4991-98d4-3ca8397d81ad" containerName="payload-processing" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301489 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="20615658-7a77-4d68-b8e6-5720a7a1f6aa" containerName="payload-processing" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301503 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eff729-76d4-41a0-a88c-5cf03cce0537" containerName="payload-processing" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301513 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6705ac81-8c30-4af3-85c0-195703dc08e7" containerName="istio-proxy" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301525 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52b3ede-52ee-460a-a7c1-c86e0389f4d4" containerName="payload-pre-processing" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301533 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e8cafb-cbb3-4c40-964a-09a4b1ee72b1" containerName="payload-processing" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301545 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92ba181-03a2-43ca-8830-3f28285dd7ca" containerName="payload-pre-processing" Nov 01 20:39:40.301823 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.301555 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3b044f-05e5-4d1d-bdf2-c7726bcaaa28" containerName="istio-proxy" Nov 01 20:39:40.306290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.306265 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.360716 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.360687 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg"] Nov 01 20:39:40.405054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.405016 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:39:40.405054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.405016 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:39:40.405283 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.405015 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:39:40.407028 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.407005 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:39:40.409853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.409817 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.410007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.409953 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.410007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.409984 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qptj\" (UniqueName: \"kubernetes.io/projected/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-kube-api-access-7qptj\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.467650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.467607 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20615658-7a77-4d68-b8e6-5720a7a1f6aa" path="/var/lib/kubelet/pods/20615658-7a77-4d68-b8e6-5720a7a1f6aa/volumes" Nov 01 20:39:40.468151 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.468130 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52b3ede-52ee-460a-a7c1-c86e0389f4d4" path="/var/lib/kubelet/pods/d52b3ede-52ee-460a-a7c1-c86e0389f4d4/volumes" Nov 01 20:39:40.510962 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.510929 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.511174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.511070 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.511174 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.511119 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qptj\" (UniqueName: \"kubernetes.io/projected/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-kube-api-access-7qptj\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.511725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.511705 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.513710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.513687 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.567306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.567236 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qptj\" (UniqueName: \"kubernetes.io/projected/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-kube-api-access-7qptj\") pod \"maas-api-revoke-sub-8678da75-xzbxg\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.620197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.620165 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:40.996517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:40.996487 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg"] Nov 01 20:39:40.998519 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:39:40.998492 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbea7954_e0d1_476c_b3d4_48262b4c8ff8.slice/crio-3768aaa5f3d7c4c8dd238c12bb55083ab0797c572e61760947fbd591b3eb83ab WatchSource:0}: Error finding container 3768aaa5f3d7c4c8dd238c12bb55083ab0797c572e61760947fbd591b3eb83ab: Status 404 returned error can't find the container with id 3768aaa5f3d7c4c8dd238c12bb55083ab0797c572e61760947fbd591b3eb83ab Nov 01 20:39:41.380025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:41.379988 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:41.621386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:41.621277 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" event={"ID":"cbea7954-e0d1-476c-b3d4-48262b4c8ff8","Type":"ContainerStarted","Data":"cd2159612f313bad6b4599989a7f7ea20281ef387770169ce574711df889d9ca"} Nov 01 20:39:41.621540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:41.621403 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" event={"ID":"cbea7954-e0d1-476c-b3d4-48262b4c8ff8","Type":"ContainerStarted","Data":"3768aaa5f3d7c4c8dd238c12bb55083ab0797c572e61760947fbd591b3eb83ab"} Nov 01 20:39:41.734596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:41.734543 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" podStartSLOduration=1.7345280060000001 podStartE2EDuration="1.734528006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:41.732751144 +0000 UTC m=+2189.407010402" watchObservedRunningTime="2026-11-01 20:39:41.734528006 +0000 UTC m=+2189.408787260" Nov 01 20:39:42.405405 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.405378 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:39:42.540947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.540849 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-credential-socket\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.540947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.540890 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-token\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-token\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.540947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.540936 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f85df6b-1a70-4df2-9c00-3240850c781d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f85df6b-1a70-4df2-9c00-3240850c781d-istiod-ca-cert\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.541260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541007 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-data\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.541260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541119 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-podinfo\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.541260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541135 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-credential-socket" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.541260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541154 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-envoy\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.541260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541231 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-kube-api-access-fd925\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-kube-api-access-fd925\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.541615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541366 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-data" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.541615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541469 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f85df6b-1a70-4df2-9c00-3240850c781d-istiod-ca-cert" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:42.541735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541623 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-certs\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.541735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.541684 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-socket\") pod \"8f85df6b-1a70-4df2-9c00-3240850c781d\" (UID: \"8f85df6b-1a70-4df2-9c00-3240850c781d\") " Nov 01 20:39:42.542205 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.542178 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-certs" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.542368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.542300 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.542368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.542320 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.542368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.542365 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f85df6b-1a70-4df2-9c00-3240850c781d-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.542493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.542378 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.542493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.542386 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-socket" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.543476 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.543453 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-token" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:42.543574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.543529 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-podinfo" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:42.544027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.544007 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-envoy" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.544120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.544068 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-kube-api-access-fd925" pod "8f85df6b-1a70-4df2-9c00-3240850c781d" (UID: "8f85df6b-1a70-4df2-9c00-3240850c781d"). InnerVolumeSpecName "kube-api-access-fd925". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:42.635178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.635139 5633 generic.go:356] "Generic (PLEG): container finished" podID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerID="0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399" exitCode=0 Nov 01 20:39:42.635903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.635222 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" event={"ID":"8f85df6b-1a70-4df2-9c00-3240850c781d","Type":"ContainerDied","Data":"0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399"} Nov 01 20:39:42.635903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.635258 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" Nov 01 20:39:42.635903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.635280 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m" event={"ID":"8f85df6b-1a70-4df2-9c00-3240850c781d","Type":"ContainerDied","Data":"2845236af244117af9ef3e9fbac64b1fbcccfa62bb39557a8df2284e8784001e"} Nov 01 20:39:42.635903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.635305 5633 scope.go:122] "RemoveContainer" containerID="0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399" Nov 01 20:39:42.640368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.639648 5633 generic.go:356] "Generic (PLEG): container finished" podID="cbea7954-e0d1-476c-b3d4-48262b4c8ff8" containerID="cd2159612f313bad6b4599989a7f7ea20281ef387770169ce574711df889d9ca" exitCode=0 Nov 01 20:39:42.640368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.639770 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" event={"ID":"cbea7954-e0d1-476c-b3d4-48262b4c8ff8","Type":"ContainerDied","Data":"cd2159612f313bad6b4599989a7f7ea20281ef387770169ce574711df889d9ca"} Nov 01 20:39:42.643621 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.643590 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.643728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.643627 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.643728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.643643 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.643728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.643657 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f85df6b-1a70-4df2-9c00-3240850c781d-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.643728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.643671 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd925\" (UniqueName: \"kubernetes.io/projected/8f85df6b-1a70-4df2-9c00-3240850c781d-kube-api-access-fd925\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.658929 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.658867 5633 scope.go:122] "RemoveContainer" containerID="0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399" Nov 01 20:39:42.659244 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:42.659221 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399\": container with ID starting with 0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399 not found: ID does not exist" containerID="0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399" Nov 01 20:39:42.659350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.659255 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399"} err="failed to get container status \"0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399\": rpc error: code = NotFound desc = could not find container \"0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399\": container with ID starting with 0ea968f5a21e0bc631622183a07cadb15f143ec30108d97dd5edbc1c1ab3f399 not found: ID does not exist" Nov 01 20:39:42.803571 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.803473 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m"] Nov 01 20:39:42.811519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:42.811486 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-4rm2m"] Nov 01 20:39:43.803053 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.803024 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:43.963631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.963597 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-xks-ca\") pod \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " Nov 01 20:39:43.963868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.963688 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-openshift-ca\") pod \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " Nov 01 20:39:43.963868 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.963748 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-kube-api-access-7qptj\" (UniqueName: \"kubernetes.io/projected/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-kube-api-access-7qptj\") pod \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\" (UID: \"cbea7954-e0d1-476c-b3d4-48262b4c8ff8\") " Nov 01 20:39:43.964087 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.964049 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-openshift-ca" pod "cbea7954-e0d1-476c-b3d4-48262b4c8ff8" (UID: "cbea7954-e0d1-476c-b3d4-48262b4c8ff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:43.964364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.964304 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:43.966086 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.966060 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-kube-api-access-7qptj" pod "cbea7954-e0d1-476c-b3d4-48262b4c8ff8" (UID: "cbea7954-e0d1-476c-b3d4-48262b4c8ff8"). InnerVolumeSpecName "kube-api-access-7qptj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:43.966164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:43.966108 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-xks-ca" pod "cbea7954-e0d1-476c-b3d4-48262b4c8ff8" (UID: "cbea7954-e0d1-476c-b3d4-48262b4c8ff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:39:44.065922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.065875 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:44.065922 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.065910 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qptj\" (UniqueName: \"kubernetes.io/projected/cbea7954-e0d1-476c-b3d4-48262b4c8ff8-kube-api-access-7qptj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:44.467589 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.467561 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" path="/var/lib/kubelet/pods/8f85df6b-1a70-4df2-9c00-3240850c781d/volumes" Nov 01 20:39:44.609655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.609617 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b"] Nov 01 20:39:44.656109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.656081 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" Nov 01 20:39:44.656261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.656125 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg" event={"ID":"cbea7954-e0d1-476c-b3d4-48262b4c8ff8","Type":"ContainerDied","Data":"3768aaa5f3d7c4c8dd238c12bb55083ab0797c572e61760947fbd591b3eb83ab"} Nov 01 20:39:44.656261 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.656172 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3768aaa5f3d7c4c8dd238c12bb55083ab0797c572e61760947fbd591b3eb83ab" Nov 01 20:39:44.659937 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.659914 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x"] Nov 01 20:39:44.771401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:44.771298 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:46.086273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:46.086219 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:46.174877 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:46.174837 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:39:46.619590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:46.619549 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:50.782478 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:50.782432 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:52.271707 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:52.271672 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652"] Nov 01 20:39:52.272117 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:52.271962 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" containerID="cri-o://c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a" gracePeriod=30 Nov 01 20:39:52.301773 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:52.301731 5633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:39:52.301967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:52.301813 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:39:52.520889 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:52.520848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:52.640272 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:52.640184 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:53.127490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.127445 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:53.146123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146091 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr"] Nov 01 20:39:53.146717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146703 5633 cpu_manager.go:414] "containerMap: removing container" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" Nov 01 20:39:53.146776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146718 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" Nov 01 20:39:53.146776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146734 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" Nov 01 20:39:53.146776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146739 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" Nov 01 20:39:53.146776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146771 5633 cpu_manager.go:414] "containerMap: removing container" podUID="cbea7954-e0d1-476c-b3d4-48262b4c8ff8" containerName="revoke-keys" Nov 01 20:39:53.146776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146777 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbea7954-e0d1-476c-b3d4-48262b4c8ff8" containerName="revoke-keys" Nov 01 20:39:53.146946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146856 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbea7954-e0d1-476c-b3d4-48262b4c8ff8" containerName="revoke-keys" Nov 01 20:39:53.146946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146867 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="841cb4b8-f821-4cf4-9aa7-2b4ad71106ca" containerName="istio-proxy" Nov 01 20:39:53.146946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.146874 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f85df6b-1a70-4df2-9c00-3240850c781d" containerName="istio-proxy" Nov 01 20:39:53.151612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.151595 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.181590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.181554 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxlz\" (UniqueName: \"kubernetes.io/projected/4d936d7e-78b5-4fb4-bb07-0009220fdd78-kube-api-access-dbxlz\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.181765 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.181701 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.181843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.181828 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.209281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.207647 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:39:53.209281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.207782 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:39:53.209281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.207877 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:39:53.214725 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.214695 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr"] Nov 01 20:39:53.256229 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.256202 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:39:53.282736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.282707 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxlz\" (UniqueName: \"kubernetes.io/projected/4d936d7e-78b5-4fb4-bb07-0009220fdd78-kube-api-access-dbxlz\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.283120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.282813 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.283120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.282928 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.283527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.283495 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.285465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.285444 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.338428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.338393 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxlz\" (UniqueName: \"kubernetes.io/projected/4d936d7e-78b5-4fb4-bb07-0009220fdd78-kube-api-access-dbxlz\") pod \"maas-api-revoke-sub-53c0a576-hr7gr\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.465610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.465584 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:53.915080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:53.915032 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr"] Nov 01 20:39:54.070752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:54.070715 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:39:54.741343 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:54.741281 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" event={"ID":"4d936d7e-78b5-4fb4-bb07-0009220fdd78","Type":"ContainerStarted","Data":"d08721b374de749b15350aebe82452309647341cf77739b1302361de572688a8"} Nov 01 20:39:54.741874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:54.741368 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" event={"ID":"4d936d7e-78b5-4fb4-bb07-0009220fdd78","Type":"ContainerStarted","Data":"f94cc9bfc8519ee07dcaac2c043ecf8999ba739635b5a38d03cb4ba6af317294"} Nov 01 20:39:54.855503 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:54.855449 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" podStartSLOduration=1.8554328770000001 podStartE2EDuration="1.855432877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:54.853716874 +0000 UTC m=+2202.527976143" watchObservedRunningTime="2026-11-01 20:39:54.855432877 +0000 UTC m=+2202.529692132" Nov 01 20:39:55.753996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:55.753952 5633 generic.go:356] "Generic (PLEG): container finished" podID="4d936d7e-78b5-4fb4-bb07-0009220fdd78" containerID="d08721b374de749b15350aebe82452309647341cf77739b1302361de572688a8" exitCode=0 Nov 01 20:39:55.754536 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:55.754034 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" event={"ID":"4d936d7e-78b5-4fb4-bb07-0009220fdd78","Type":"ContainerDied","Data":"d08721b374de749b15350aebe82452309647341cf77739b1302361de572688a8"} Nov 01 20:39:55.928234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:55.928197 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:56.930761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:56.930737 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:57.045845 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.045759 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-openshift-ca\") pod \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " Nov 01 20:39:57.046030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.045907 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d936d7e-78b5-4fb4-bb07-0009220fdd78-kube-api-access-dbxlz\" (UniqueName: \"kubernetes.io/projected/4d936d7e-78b5-4fb4-bb07-0009220fdd78-kube-api-access-dbxlz\") pod \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " Nov 01 20:39:57.046030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.045998 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-xks-ca\") pod \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\" (UID: \"4d936d7e-78b5-4fb4-bb07-0009220fdd78\") " Nov 01 20:39:57.046244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.046216 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-openshift-ca" pod "4d936d7e-78b5-4fb4-bb07-0009220fdd78" (UID: "4d936d7e-78b5-4fb4-bb07-0009220fdd78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:57.046477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.046457 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.048909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.048880 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-xks-ca" pod "4d936d7e-78b5-4fb4-bb07-0009220fdd78" (UID: "4d936d7e-78b5-4fb4-bb07-0009220fdd78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:39:57.049134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.049111 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d936d7e-78b5-4fb4-bb07-0009220fdd78-kube-api-access-dbxlz" pod "4d936d7e-78b5-4fb4-bb07-0009220fdd78" (UID: "4d936d7e-78b5-4fb4-bb07-0009220fdd78"). InnerVolumeSpecName "kube-api-access-dbxlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:57.147883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.147849 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbxlz\" (UniqueName: \"kubernetes.io/projected/4d936d7e-78b5-4fb4-bb07-0009220fdd78-kube-api-access-dbxlz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.147883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.147881 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d936d7e-78b5-4fb4-bb07-0009220fdd78-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.538776 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.538750 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:39:57.653577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.653494 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-envoy\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.653577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.653535 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-kube-api-access-ms79n\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-kube-api-access-ms79n\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.653797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.653624 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-token\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-token\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.653797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.653663 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-credential-socket\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.653940 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.653826 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-podinfo\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.654004 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.653974 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-credential-socket" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:57.654061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.653997 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-socket\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.654061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654050 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-data\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.654184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654102 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-certs\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.654184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654177 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4deeb6d-3552-4d30-8e78-8e848757a623-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4deeb6d-3552-4d30-8e78-8e848757a623-istiod-ca-cert\") pod \"c4deeb6d-3552-4d30-8e78-8e848757a623\" (UID: \"c4deeb6d-3552-4d30-8e78-8e848757a623\") " Nov 01 20:39:57.654294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654241 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-socket" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:57.654559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654315 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-data" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:57.654559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654409 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-certs" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:57.654688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654603 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4deeb6d-3552-4d30-8e78-8e848757a623-istiod-ca-cert" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:57.654819 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654796 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4deeb6d-3552-4d30-8e78-8e848757a623-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.654886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654821 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.654886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654832 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.654886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654840 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.654886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.654849 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.656109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.656088 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-kube-api-access-ms79n" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "kube-api-access-ms79n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:57.656243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.656188 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-token" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:57.656537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.656516 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-envoy" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:57.656589 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.656518 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-podinfo" pod "c4deeb6d-3552-4d30-8e78-8e848757a623" (UID: "c4deeb6d-3552-4d30-8e78-8e848757a623"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:57.756607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.756567 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.756607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.756600 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms79n\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-kube-api-access-ms79n\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.756607 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.756617 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.756874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.756625 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4deeb6d-3552-4d30-8e78-8e848757a623-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:57.771623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.771587 5633 generic.go:356] "Generic (PLEG): container finished" podID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerID="c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a" exitCode=0 Nov 01 20:39:57.771805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.771632 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" Nov 01 20:39:57.771805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.771659 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" event={"ID":"c4deeb6d-3552-4d30-8e78-8e848757a623","Type":"ContainerDied","Data":"c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a"} Nov 01 20:39:57.771805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.771714 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652" event={"ID":"c4deeb6d-3552-4d30-8e78-8e848757a623","Type":"ContainerDied","Data":"252e5c54477ff308212208176f7914f759c9954bbd9b94552684442e53182521"} Nov 01 20:39:57.771805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.771739 5633 scope.go:122] "RemoveContainer" containerID="c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a" Nov 01 20:39:57.773735 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.773707 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" event={"ID":"4d936d7e-78b5-4fb4-bb07-0009220fdd78","Type":"ContainerDied","Data":"f94cc9bfc8519ee07dcaac2c043ecf8999ba739635b5a38d03cb4ba6af317294"} Nov 01 20:39:57.773834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.773764 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f94cc9bfc8519ee07dcaac2c043ecf8999ba739635b5a38d03cb4ba6af317294" Nov 01 20:39:57.773834 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.773826 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr" Nov 01 20:39:57.784548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.784531 5633 scope.go:122] "RemoveContainer" containerID="c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a" Nov 01 20:39:57.785243 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:39:57.785206 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a\": container with ID starting with c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a not found: ID does not exist" containerID="c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a" Nov 01 20:39:57.785383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.785247 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a"} err="failed to get container status \"c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a\": rpc error: code = NotFound desc = could not find container \"c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a\": container with ID starting with c76fa2258c3a2ddc9f1c957f82e9c69451699a28ed825148861e2bdcda5b745a not found: ID does not exist" Nov 01 20:39:57.989984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.989944 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652"] Nov 01 20:39:57.993667 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:57.993641 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-p6652"] Nov 01 20:39:58.467635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:58.467591 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" path="/var/lib/kubelet/pods/c4deeb6d-3552-4d30-8e78-8e848757a623/volumes" Nov 01 20:39:58.637707 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:58.637669 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:39:59.631222 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.631176 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" containerID="cri-o://051f35b43cd3bed88b42dd5311ad6d73e9726b11e80b8cbbe464e66aee942c45" gracePeriod=45 Nov 01 20:39:59.681417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.681366 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" containerID="cri-o://577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898" gracePeriod=45 Nov 01 20:39:59.783302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.783257 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": dial tcp 10.134.0.81:8000: connect: connection refused" Nov 01 20:39:59.791991 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.791959 5633 generic.go:356] "Generic (PLEG): container finished" podID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerID="051f35b43cd3bed88b42dd5311ad6d73e9726b11e80b8cbbe464e66aee942c45" exitCode=0 Nov 01 20:39:59.792136 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.792025 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" event={"ID":"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45","Type":"ContainerDied","Data":"051f35b43cd3bed88b42dd5311ad6d73e9726b11e80b8cbbe464e66aee942c45"} Nov 01 20:39:59.807971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.807928 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": dial tcp 10.134.0.91:8000: connect: connection refused" Nov 01 20:39:59.965268 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.965242 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:39:59.979031 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:39:59.979003 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089061 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-kube-api-access-z4ncj\" (UniqueName: \"kubernetes.io/projected/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-kube-api-access-z4ncj\") pod \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089120 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f1de7c-e045-4fb6-98e4-97972982358c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f1de7c-e045-4fb6-98e4-97972982358c-tls-certs\") pod \"b4f1de7c-e045-4fb6-98e4-97972982358c\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089179 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-dshm\") pod \"b4f1de7c-e045-4fb6-98e4-97972982358c\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089242 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tls-certs\") pod \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089303 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-tmp-dir\") pod \"b4f1de7c-e045-4fb6-98e4-97972982358c\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089378 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-home\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-home\") pod \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089412 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-model-cache\") pod \"b4f1de7c-e045-4fb6-98e4-97972982358c\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089439 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tmp-dir\") pod \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089513 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-model-cache\") pod \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089568 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-home\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-home\") pod \"b4f1de7c-e045-4fb6-98e4-97972982358c\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089674 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-dshm\") pod \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\" (UID: \"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45\") " Nov 01 20:40:00.092827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.089724 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f1de7c-e045-4fb6-98e4-97972982358c-kube-api-access-tdw8k\" (UniqueName: \"kubernetes.io/projected/b4f1de7c-e045-4fb6-98e4-97972982358c-kube-api-access-tdw8k\") pod \"b4f1de7c-e045-4fb6-98e4-97972982358c\" (UID: \"b4f1de7c-e045-4fb6-98e4-97972982358c\") " Nov 01 20:40:00.096546 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.096506 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-home" pod "b4f1de7c-e045-4fb6-98e4-97972982358c" (UID: "b4f1de7c-e045-4fb6-98e4-97972982358c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.096709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.096681 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-model-cache" pod "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" (UID: "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.096882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.096854 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-model-cache" pod "b4f1de7c-e045-4fb6-98e4-97972982358c" (UID: "b4f1de7c-e045-4fb6-98e4-97972982358c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.097043 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.097021 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tmp-dir" pod "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" (UID: "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.097507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.097480 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-home" pod "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" (UID: "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.098386 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.098356 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-dshm" pod "b4f1de7c-e045-4fb6-98e4-97972982358c" (UID: "b4f1de7c-e045-4fb6-98e4-97972982358c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.099241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.099035 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-dshm" pod "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" (UID: "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.099715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.099399 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-tmp-dir" pod "b4f1de7c-e045-4fb6-98e4-97972982358c" (UID: "b4f1de7c-e045-4fb6-98e4-97972982358c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.100102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.100071 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tls-certs" pod "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" (UID: "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:40:00.100774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.100741 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f1de7c-e045-4fb6-98e4-97972982358c-kube-api-access-tdw8k" pod "b4f1de7c-e045-4fb6-98e4-97972982358c" (UID: "b4f1de7c-e045-4fb6-98e4-97972982358c"). InnerVolumeSpecName "kube-api-access-tdw8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:00.101234 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.101206 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f1de7c-e045-4fb6-98e4-97972982358c-tls-certs" pod "b4f1de7c-e045-4fb6-98e4-97972982358c" (UID: "b4f1de7c-e045-4fb6-98e4-97972982358c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:40:00.102135 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.102109 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-kube-api-access-z4ncj" pod "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" (UID: "ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45"). InnerVolumeSpecName "kube-api-access-z4ncj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:00.196995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.196908 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4ncj\" (UniqueName: \"kubernetes.io/projected/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-kube-api-access-z4ncj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.196995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.196972 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f1de7c-e045-4fb6-98e4-97972982358c-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.196995 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.196988 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197002 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197013 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197024 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197035 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197048 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197059 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197070 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f1de7c-e045-4fb6-98e4-97972982358c-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197081 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.197277 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.197092 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdw8k\" (UniqueName: \"kubernetes.io/projected/b4f1de7c-e045-4fb6-98e4-97972982358c-kube-api-access-tdw8k\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.287428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.287396 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:00.800539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.800452 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" Nov 01 20:40:00.801025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.800453 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b" event={"ID":"ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45","Type":"ContainerDied","Data":"a667a216a57a154095157f5cb75e71c46ae34c4c84ac358ff52b34184e87f7fc"} Nov 01 20:40:00.801025 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.800610 5633 scope.go:122] "RemoveContainer" containerID="051f35b43cd3bed88b42dd5311ad6d73e9726b11e80b8cbbe464e66aee942c45" Nov 01 20:40:00.802147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.802122 5633 generic.go:356] "Generic (PLEG): container finished" podID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerID="577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898" exitCode=0 Nov 01 20:40:00.802220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.802177 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" event={"ID":"b4f1de7c-e045-4fb6-98e4-97972982358c","Type":"ContainerDied","Data":"577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898"} Nov 01 20:40:00.802220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.802205 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" event={"ID":"b4f1de7c-e045-4fb6-98e4-97972982358c","Type":"ContainerDied","Data":"3764a7503b21d00ff33777ab79a40fead515ad25db3ed038c66744a17f9868f5"} Nov 01 20:40:00.802220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.802177 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x" Nov 01 20:40:00.811186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.811163 5633 scope.go:122] "RemoveContainer" containerID="577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898" Nov 01 20:40:00.825119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.825093 5633 scope.go:122] "RemoveContainer" containerID="577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898" Nov 01 20:40:00.825646 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:00.825615 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898\": container with ID starting with 577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898 not found: ID does not exist" containerID="577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898" Nov 01 20:40:00.825786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.825662 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898"} err="failed to get container status \"577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898\": rpc error: code = NotFound desc = could not find container \"577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898\": container with ID starting with 577296e3f13e265b49c285095e22b8f84efd5422b3e56b4940b0087dc7560898 not found: ID does not exist" Nov 01 20:40:00.965253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:00.965211 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x"] Nov 01 20:40:01.014558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:01.014519 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-w3-1b1134-kserve-7c5d6c4b88-glm7x"] Nov 01 20:40:01.122178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:01.122090 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b"] Nov 01 20:40:01.125565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:01.125542 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/distinct-2-w3-1b1134-kserve-6bb7985b59-n2s8b"] Nov 01 20:40:01.908177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:01.908138 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:02.128690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:02.128653 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:02.343865 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:02.343827 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:02.468963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:02.468928 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" path="/var/lib/kubelet/pods/ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45/volumes" Nov 01 20:40:02.469426 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:02.469412 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" path="/var/lib/kubelet/pods/b4f1de7c-e045-4fb6-98e4-97972982358c/volumes" Nov 01 20:40:03.376129 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:03.376093 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:03.870705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:03.870668 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:03.895317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:03.895286 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6"] Nov 01 20:40:03.895585 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:03.895557 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" containerID="cri-o://6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6" gracePeriod=30 Nov 01 20:40:04.618394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:04.618353 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:04.815249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:04.815204 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:04.955242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:04.955163 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:05.227615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:05.227582 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:07.338838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:07.338792 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:09.157092 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.157063 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:40:09.234123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234090 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-envoy\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234161 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-credential-socket\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234208 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-certs\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234291 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e7055e-4677-478f-bb8e-4423b79d72c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e7055e-4677-478f-bb8e-4423b79d72c0-istiod-ca-cert\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234322 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-socket\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234448 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-podinfo\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234505 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-token\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-token\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234528 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-credential-socket" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:09.234657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234575 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-kube-api-access-x8bp8\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-kube-api-access-x8bp8\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234599 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-socket" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:09.234657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234627 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-data\") pod \"76e7055e-4677-478f-bb8e-4423b79d72c0\" (UID: \"76e7055e-4677-478f-bb8e-4423b79d72c0\") " Nov 01 20:40:09.234990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234677 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e7055e-4677-478f-bb8e-4423b79d72c0-istiod-ca-cert" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:40:09.234990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234676 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-certs" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:09.234990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.234894 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-data" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:09.235186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.235168 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.235265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.235189 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.235265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.235204 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76e7055e-4677-478f-bb8e-4423b79d72c0-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.235265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.235220 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.235265 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.235233 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.236673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.236648 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-envoy" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:09.237019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.237001 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-kube-api-access-x8bp8" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "kube-api-access-x8bp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:09.237371 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.237352 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-podinfo" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:40:09.237451 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.237376 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-token" pod "76e7055e-4677-478f-bb8e-4423b79d72c0" (UID: "76e7055e-4677-478f-bb8e-4423b79d72c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:09.336993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.336955 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.336993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.336985 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.336993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.336997 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8bp8\" (UniqueName: \"kubernetes.io/projected/76e7055e-4677-478f-bb8e-4423b79d72c0-kube-api-access-x8bp8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.337323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.337006 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76e7055e-4677-478f-bb8e-4423b79d72c0-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:09.892325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.892292 5633 generic.go:356] "Generic (PLEG): container finished" podID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerID="6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6" exitCode=0 Nov 01 20:40:09.892531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.892374 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" Nov 01 20:40:09.892531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.892379 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" event={"ID":"76e7055e-4677-478f-bb8e-4423b79d72c0","Type":"ContainerDied","Data":"6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6"} Nov 01 20:40:09.892531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.892443 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6" event={"ID":"76e7055e-4677-478f-bb8e-4423b79d72c0","Type":"ContainerDied","Data":"90a2035e66dd25e5a997a753771dd5c9218d6f398e2a9a2dc2621465308cc0aa"} Nov 01 20:40:09.892531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.892466 5633 scope.go:122] "RemoveContainer" containerID="6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6" Nov 01 20:40:09.904171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.904150 5633 scope.go:122] "RemoveContainer" containerID="fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7" Nov 01 20:40:09.914242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.914219 5633 scope.go:122] "RemoveContainer" containerID="6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6" Nov 01 20:40:09.914548 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:09.914520 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6\": container with ID starting with 6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6 not found: ID does not exist" containerID="6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6" Nov 01 20:40:09.914600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.914561 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6"} err="failed to get container status \"6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6\": rpc error: code = NotFound desc = could not find container \"6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6\": container with ID starting with 6e0946684496f802b874e053c826c56712c09dba5a9482388bd99ac91548d5c6 not found: ID does not exist" Nov 01 20:40:09.914600 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.914583 5633 scope.go:122] "RemoveContainer" containerID="fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7" Nov 01 20:40:09.914853 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:09.914827 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7\": container with ID starting with fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7 not found: ID does not exist" containerID="fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7" Nov 01 20:40:09.914853 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:09.914847 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7"} err="failed to get container status \"fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7\": rpc error: code = NotFound desc = could not find container \"fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7\": container with ID starting with fee7a0a22dc169e54f581e4128cdcc06283636cc2acd6eca354ff8d2eb4291c7 not found: ID does not exist" Nov 01 20:40:10.110622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:10.106896 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6"] Nov 01 20:40:10.154681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:10.154599 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-13cbc9-openshift-default-7d44fd9b9c-fgsf6"] Nov 01 20:40:10.468391 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:10.468360 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" path="/var/lib/kubelet/pods/76e7055e-4677-478f-bb8e-4423b79d72c0/volumes" Nov 01 20:40:10.813568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:10.813486 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:10.941198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:10.941141 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:11.070029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:11.069933 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:11.394978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:11.394895 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:12.181714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:12.181679 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:16.157409 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:16.157370 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:40:16.883690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:16.883650 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:17.118548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.118510 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:17.277307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.277269 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:17.367917 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.367889 5633 scope.go:122] "RemoveContainer" containerID="bdd237a1a981cbb385b22231836795cc94a3eb52e3146cdf8b2f40e7faee1f1e" Nov 01 20:40:17.378374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.378354 5633 scope.go:122] "RemoveContainer" containerID="09e896110dd1749f62670e3d9fdc44898ae7f0b8315b7d5593687c4e1b75c06b" Nov 01 20:40:17.388622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.388603 5633 scope.go:122] "RemoveContainer" containerID="06fd2621a3e1c9351816b46d30a26d3919b7c81a48eea25499f730d26a86dbdc" Nov 01 20:40:17.398545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.398524 5633 scope.go:122] "RemoveContainer" containerID="a8f0fbb4940e9074d79b617b8172525a33580817a0f5c223a210e1126eb41774" Nov 01 20:40:17.408427 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.408395 5633 scope.go:122] "RemoveContainer" containerID="6a25274d6e8b71d57a80e643fb6b3319a514c5a8387968200862087a48349a0f" Nov 01 20:40:17.419459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:17.419437 5633 scope.go:122] "RemoveContainer" containerID="182c35002786ec3296ae7976eb37ab5be3e8f2bc9a675772dbc67aff3e199f7e" Nov 01 20:40:18.744172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:18.744125 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:21.596472 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:21.596428 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:21.731225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:21.731187 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:22.650821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:22.650786 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:23.235242 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:23.235207 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:40:23.901029 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:23.900907 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:24.025179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:24.025130 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:25.311052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:25.310928 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:25.692120 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:25.690772 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:27.412975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:27.412941 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:27.791755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:27.791716 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:27.913769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:27.913727 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:28.325599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:28.325566 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:28.457295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:28.457258 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:29.348960 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:29.348924 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:29.807262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:29.807224 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7"] Nov 01 20:40:29.807924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:29.807500 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" containerID="cri-o://309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e" gracePeriod=30 Nov 01 20:40:29.837101 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:29.837067 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:30.184317 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:30.184223 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:30.304828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:30.304776 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:30.953605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:30.953567 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:31.133199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:31.133155 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:32.409245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:32.409210 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:32.545091 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:32.545056 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-dc7c20a2c19f: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:32.545091 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:32.545087 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:32.545390 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:32.545161 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials podName:b1771e1a-c04c-4257-8cf6-deb67eefa7ac nodeName:}" failed. No retries permitted until 2026-11-01 20:40:33.045138965 +0000 UTC m=+2240.719398200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac") : secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:33.051138 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:33.051103 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-dc7c20a2c19f: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:33.051138 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:33.051137 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:33.051395 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:33.051189 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials podName:b1771e1a-c04c-4257-8cf6-deb67eefa7ac nodeName:}" failed. No retries permitted until 2026-11-01 20:40:34.05117682 +0000 UTC m=+2241.725436054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac") : secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:34.064105 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:34.064073 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-dc7c20a2c19f: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:34.064105 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:34.064105 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:34.064785 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:34.064169 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials podName:b1771e1a-c04c-4257-8cf6-deb67eefa7ac nodeName:}" failed. No retries permitted until 2026-11-01 20:40:36.064151214 +0000 UTC m=+2243.738410448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac") : secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:35.084463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.084381 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:40:35.103640 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.103608 5633 generic.go:356] "Generic (PLEG): container finished" podID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerID="309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e" exitCode=0 Nov 01 20:40:35.103839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.103661 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" event={"ID":"9070b4cc-cc17-4396-9a63-83963140bcc8","Type":"ContainerDied","Data":"309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e"} Nov 01 20:40:35.103839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.103706 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" event={"ID":"9070b4cc-cc17-4396-9a63-83963140bcc8","Type":"ContainerDied","Data":"f50a3cd29300830f3a74c1485ae9d8228274e759412d0ee508e926f46064321e"} Nov 01 20:40:35.103839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.103728 5633 scope.go:122] "RemoveContainer" containerID="309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e" Nov 01 20:40:35.103839 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.103661 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7" Nov 01 20:40:35.113933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.113909 5633 scope.go:122] "RemoveContainer" containerID="309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e" Nov 01 20:40:35.114234 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:35.114214 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e\": container with ID starting with 309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e not found: ID does not exist" containerID="309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e" Nov 01 20:40:35.114295 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.114245 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e"} err="failed to get container status \"309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e\": rpc error: code = NotFound desc = could not find container \"309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e\": container with ID starting with 309b8ea6b1259aac10b59008a82567bb0d7c9fc57103979aa128173c0e4fe94e not found: ID does not exist" Nov 01 20:40:35.178240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178196 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-token\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-token\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178266 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-socket\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178471 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178303 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-data\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178483 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-kube-api-access-chbwc\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-kube-api-access-chbwc\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178580 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178551 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-podinfo\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178596 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-socket" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.178705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178668 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-data" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.178705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178682 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-certs\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178733 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-envoy\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178758 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9070b4cc-cc17-4396-9a63-83963140bcc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9070b4cc-cc17-4396-9a63-83963140bcc8-istiod-ca-cert\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178778 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-credential-socket\") pod \"9070b4cc-cc17-4396-9a63-83963140bcc8\" (UID: \"9070b4cc-cc17-4396-9a63-83963140bcc8\") " Nov 01 20:40:35.178885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.178871 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-certs" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.179273 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.179159 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9070b4cc-cc17-4396-9a63-83963140bcc8-istiod-ca-cert" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:40:35.179433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.179324 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-credential-socket" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.179533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.179513 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.179605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.179539 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.179605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.179553 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.179605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.179568 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9070b4cc-cc17-4396-9a63-83963140bcc8-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.179605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.179583 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.180879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.180847 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-token" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:35.181000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.180961 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-kube-api-access-chbwc" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "kube-api-access-chbwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:35.181348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.181314 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-envoy" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.181348 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.181317 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-podinfo" pod "9070b4cc-cc17-4396-9a63-83963140bcc8" (UID: "9070b4cc-cc17-4396-9a63-83963140bcc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:40:35.281543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.281506 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chbwc\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-kube-api-access-chbwc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.281543 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.281544 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.281832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.281559 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.281832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.281572 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9070b4cc-cc17-4396-9a63-83963140bcc8-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.525456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.525410 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7"] Nov 01 20:40:35.530964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:35.530934 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-kb7d7"] Nov 01 20:40:36.091649 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:36.091615 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-dc7c20a2c19f: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:36.091649 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:36.091645 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:36.092115 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:36.091701 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials podName:b1771e1a-c04c-4257-8cf6-deb67eefa7ac nodeName:}" failed. No retries permitted until 2026-11-01 20:40:40.091687367 +0000 UTC m=+2247.765946601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac") : secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:36.106442 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:36.106409 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:36.468122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:36.468086 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" path="/var/lib/kubelet/pods/9070b4cc-cc17-4396-9a63-83963140bcc8/volumes" Nov 01 20:40:36.478235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:36.478204 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:37.059429 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:37.059389 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:37.280118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:37.280081 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:40.140749 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:40.140714 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-caf6c2f34a9e: secret "agc-cred-caf6c2f34a9e" not found Nov 01 20:40:40.140749 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:40.140751 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-dc7c20a2c19f: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:40.141376 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:40.140767 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v: [secret "agc-cred-caf6c2f34a9e" not found, secret "agc-cred-dc7c20a2c19f" not found] Nov 01 20:40:40.141376 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:40.140838 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials podName:b1771e1a-c04c-4257-8cf6-deb67eefa7ac nodeName:}" failed. No retries permitted until 2026-11-01 20:40:48.140815019 +0000 UTC m=+2255.815074266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac") : [secret "agc-cred-caf6c2f34a9e" not found, secret "agc-cred-dc7c20a2c19f" not found] Nov 01 20:40:41.983699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:41.983664 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:42.948682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:42.948638 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:43.150032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:43.149995 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:43.370260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:43.370221 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:44.361193 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:44.361157 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:44.485767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:44.485734 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:46.158362 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:46.158294 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:40:46.187402 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:46.187367 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:40:46.788590 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:46.788550 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:48.155574 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:48.155535 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-caf6c2f34a9e: secret "agc-cred-caf6c2f34a9e" not found Nov 01 20:40:48.155574 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:48.155570 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-dc7c20a2c19f: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:40:48.155574 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:48.155582 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v: [secret "agc-cred-caf6c2f34a9e" not found, secret "agc-cred-dc7c20a2c19f" not found] Nov 01 20:40:48.156087 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:40:48.155648 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials podName:b1771e1a-c04c-4257-8cf6-deb67eefa7ac nodeName:}" failed. No retries permitted until 2026-11-01 20:41:04.155631141 +0000 UTC m=+2271.829890392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac") : [secret "agc-cred-caf6c2f34a9e" not found, secret "agc-cred-dc7c20a2c19f" not found] Nov 01 20:40:51.498258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:51.498171 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:56.222454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:56.222414 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:40:56.996928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:56.996893 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:40:59.779827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:40:59.779788 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:00.051139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:00.051050 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l"] Nov 01 20:41:00.100479 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:00.100443 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-734772f6-7hz4l"] Nov 01 20:41:00.468715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:00.468668 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17871a7-c69c-4a6c-a567-dbe08d346f20" path="/var/lib/kubelet/pods/d17871a7-c69c-4a6c-a567-dbe08d346f20/volumes" Nov 01 20:41:00.878593 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:00.878492 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:01.050187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:01.050148 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v"] Nov 01 20:41:04.178483 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:41:04.178452 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-caf6c2f34a9e: secret "agc-cred-caf6c2f34a9e" not found Nov 01 20:41:04.178483 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:41:04.178482 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-dc7c20a2c19f: secret "agc-cred-dc7c20a2c19f" not found Nov 01 20:41:04.178483 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:41:04.178493 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v: [secret "agc-cred-caf6c2f34a9e" not found, secret "agc-cred-dc7c20a2c19f" not found] Nov 01 20:41:04.178994 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:41:04.178547 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials podName:b1771e1a-c04c-4257-8cf6-deb67eefa7ac nodeName:}" failed. No retries permitted until 2026-11-01 20:41:36.178533076 +0000 UTC m=+2303.852792309 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac") : [secret "agc-cred-caf6c2f34a9e" not found, secret "agc-cred-dc7c20a2c19f" not found] Nov 01 20:41:05.608517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:05.608475 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:06.050902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.050840 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" podUID="b1771e1a-c04c-4257-8cf6-deb67eefa7ac" containerName="payload-processing" containerID="cri-o://6d9d1fee8b98feea902068e54fc4bb7492eab081cb7cd3c7227fe12a39ae1271" gracePeriod=25 Nov 01 20:41:06.312061 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.312031 5633 generic.go:356] "Generic (PLEG): container finished" podID="b1771e1a-c04c-4257-8cf6-deb67eefa7ac" containerID="6d9d1fee8b98feea902068e54fc4bb7492eab081cb7cd3c7227fe12a39ae1271" exitCode=0 Nov 01 20:41:06.312251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.312100 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" event={"ID":"b1771e1a-c04c-4257-8cf6-deb67eefa7ac","Type":"ContainerDied","Data":"6d9d1fee8b98feea902068e54fc4bb7492eab081cb7cd3c7227fe12a39ae1271"} Nov 01 20:41:06.312251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.312149 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" event={"ID":"b1771e1a-c04c-4257-8cf6-deb67eefa7ac","Type":"ContainerDied","Data":"9d7a94289ea28f69428fdb8b1552eb3ab6b686c40dd8366626e21c14be7cfe10"} Nov 01 20:41:06.312251 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.312165 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7a94289ea28f69428fdb8b1552eb3ab6b686c40dd8366626e21c14be7cfe10" Nov 01 20:41:06.322388 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.322366 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:41:06.403236 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.403199 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-kube-api-access-7cmhj\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-kube-api-access-7cmhj\") pod \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " Nov 01 20:41:06.403465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.403369 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-routing-overlay-volume\") pod \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " Nov 01 20:41:06.403465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.403454 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-plugins-config-volume\") pod \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " Nov 01 20:41:06.403567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.403531 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials\") pod \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\" (UID: \"b1771e1a-c04c-4257-8cf6-deb67eefa7ac\") " Nov 01 20:41:06.403744 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.403719 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-routing-overlay-volume" pod "b1771e1a-c04c-4257-8cf6-deb67eefa7ac" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:41:06.403860 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.403835 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-plugins-config-volume" pod "b1771e1a-c04c-4257-8cf6-deb67eefa7ac" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:41:06.404049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.404033 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:06.404098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.404051 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:06.405816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.405795 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-kube-api-access-7cmhj" pod "b1771e1a-c04c-4257-8cf6-deb67eefa7ac" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac"). InnerVolumeSpecName "kube-api-access-7cmhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:41:06.405919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.405838 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials" pod "b1771e1a-c04c-4257-8cf6-deb67eefa7ac" (UID: "b1771e1a-c04c-4257-8cf6-deb67eefa7ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:41:06.505635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.505599 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cmhj\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-kube-api-access-7cmhj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:06.505635 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:06.505628 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b1771e1a-c04c-4257-8cf6-deb67eefa7ac-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:07.318570 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:07.318538 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v" Nov 01 20:41:07.482026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:07.481994 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v"] Nov 01 20:41:07.530924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:07.530890 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-d76bmmc8v"] Nov 01 20:41:08.467418 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:08.467386 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1771e1a-c04c-4257-8cf6-deb67eefa7ac" path="/var/lib/kubelet/pods/b1771e1a-c04c-4257-8cf6-deb67eefa7ac/volumes" Nov 01 20:41:10.290256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:10.290212 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:14.984374 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:14.984299 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:15.945311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:15.945278 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:16.157061 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:41:16.157024 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:41:17.541292 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:17.541251 5633 scope.go:122] "RemoveContainer" containerID="6d9d1fee8b98feea902068e54fc4bb7492eab081cb7cd3c7227fe12a39ae1271" Nov 01 20:41:19.686804 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:19.686761 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:24.383609 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:24.383566 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:29.179124 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:29.179081 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:30.596866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:30.596829 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:33.883456 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:33.883416 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:38.583906 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:38.583871 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:40.258044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:40.258004 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:40.393127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:40.393087 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:43.375771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:43.375722 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:46.156766 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:41:46.156724 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:41:48.078968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:48.078937 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:48.531620 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:48.531578 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z"] Nov 01 20:41:48.531843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:48.531820 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" containerID="cri-o://7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784" gracePeriod=30 Nov 01 20:41:49.741305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:49.741260 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:41:49.741739 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:49.741368 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:41:52.799936 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:52.799897 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:41:53.799681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.799616 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:41:53.869699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.869661 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7794ca7c-889b-4042-b546-681d2182f466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7794ca7c-889b-4042-b546-681d2182f466-istiod-ca-cert\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.869719 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-socket\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.869758 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-envoy\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.869801 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-credential-socket\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.869847 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7794ca7c-889b-4042-b546-681d2182f466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7794ca7c-889b-4042-b546-681d2182f466-istio-podinfo\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.869915 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-istio-token\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-istio-token\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870048 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-certs\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870106 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-socket" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:53.870164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870113 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-data\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870193 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-kube-api-access-nt466\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-kube-api-access-nt466\") pod \"7794ca7c-889b-4042-b546-681d2182f466\" (UID: \"7794ca7c-889b-4042-b546-681d2182f466\") " Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870211 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7794ca7c-889b-4042-b546-681d2182f466-istiod-ca-cert" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870292 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-data" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870381 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-credential-socket" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870406 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-certs" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870704 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870724 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870735 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7794ca7c-889b-4042-b546-681d2182f466-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870744 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.870851 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.870753 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.872494 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.872471 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-kube-api-access-nt466" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "kube-api-access-nt466". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:41:53.872612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.872542 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7794ca7c-889b-4042-b546-681d2182f466-istio-podinfo" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:41:53.872743 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.872726 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-istio-token" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:41:53.872812 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.872734 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-envoy" pod "7794ca7c-889b-4042-b546-681d2182f466" (UID: "7794ca7c-889b-4042-b546-681d2182f466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:53.972096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.972055 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7794ca7c-889b-4042-b546-681d2182f466-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.972096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.972086 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7794ca7c-889b-4042-b546-681d2182f466-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.972096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.972102 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:53.972414 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:53.972116 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt466\" (UniqueName: \"kubernetes.io/projected/7794ca7c-889b-4042-b546-681d2182f466-kube-api-access-nt466\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:41:54.651816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.651774 5633 generic.go:356] "Generic (PLEG): container finished" podID="7794ca7c-889b-4042-b546-681d2182f466" containerID="7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784" exitCode=0 Nov 01 20:41:54.652010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.651832 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" Nov 01 20:41:54.652010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.651859 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" event={"ID":"7794ca7c-889b-4042-b546-681d2182f466","Type":"ContainerDied","Data":"7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784"} Nov 01 20:41:54.652010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.651907 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z" event={"ID":"7794ca7c-889b-4042-b546-681d2182f466","Type":"ContainerDied","Data":"3cf3c63fde88f2e9b88949a563975ca6ed4cfdd7c737507275b2930ea3e0ed78"} Nov 01 20:41:54.652010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.651927 5633 scope.go:122] "RemoveContainer" containerID="7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784" Nov 01 20:41:54.662598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.662575 5633 scope.go:122] "RemoveContainer" containerID="7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784" Nov 01 20:41:54.662940 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:41:54.662918 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784\": container with ID starting with 7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784 not found: ID does not exist" containerID="7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784" Nov 01 20:41:54.663040 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.662950 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784"} err="failed to get container status \"7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784\": rpc error: code = NotFound desc = could not find container \"7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784\": container with ID starting with 7414fe1a175d7c097337407a3802277a57eef3fcd2db1552a52ae9f952255784 not found: ID does not exist" Nov 01 20:41:54.865832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.865742 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z"] Nov 01 20:41:54.915412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:54.915373 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-sbq8z"] Nov 01 20:41:56.467400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:56.467358 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7794ca7c-889b-4042-b546-681d2182f466" path="/var/lib/kubelet/pods/7794ca7c-889b-4042-b546-681d2182f466/volumes" Nov 01 20:41:57.476038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:41:57.475997 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:02.171243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:02.171202 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:06.875311 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:06.875271 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:11.588672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:11.588626 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:16.156487 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:42:16.156444 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:42:16.302376 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:16.302311 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:17.579235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:17.579197 5633 scope.go:122] "RemoveContainer" containerID="6111bdc2da2278a05ac6674aaac4043f6990694909ccd86a9060132661bab7ad" Nov 01 20:42:17.589312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:17.589287 5633 scope.go:122] "RemoveContainer" containerID="eb55f1f8426131881d1310538e1cabdbdeed87729a41570b0bc734183ab7ad53" Nov 01 20:42:20.959008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:20.958963 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:25.684625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:25.684585 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:30.391291 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:30.391247 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:30.872698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:30.872660 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:35.080540 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:35.080505 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:39.776726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:39.776686 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:44.473778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:44.473740 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:46.053713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:46.053676 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7"] Nov 01 20:42:46.102420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:46.102385 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289dd985-nzpj7"] Nov 01 20:42:46.171486 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:42:46.171438 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:42:46.467385 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:46.467357 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a799591-35e2-4292-8f87-b1f7b6764aee" path="/var/lib/kubelet/pods/0a799591-35e2-4292-8f87-b1f7b6764aee/volumes" Nov 01 20:42:49.181701 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:49.181663 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:53.869948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:53.869910 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:42:58.045961 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:58.045915 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj"] Nov 01 20:42:58.093588 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:58.093552 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbde14f-xzwrj"] Nov 01 20:42:58.466541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:58.466508 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74" path="/var/lib/kubelet/pods/d55b4cdd-f5c5-4dc0-bb89-321fd5a64a74/volumes" Nov 01 20:42:58.584736 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:42:58.584690 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:03.295358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:03.295294 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:08.087035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:08.086992 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:12.818595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:12.818549 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:12.953114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:12.953075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:16.156145 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:43:16.156103 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:43:17.479075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:17.479033 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:18.092071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:18.092030 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:18.350175 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:18.350090 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:19.110088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:19.110055 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:22.187470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:22.187432 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:23.523549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:23.523514 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:43:25.080237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.080202 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:25.218587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.218549 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:25.335615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.335532 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf"] Nov 01 20:43:25.336122 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336108 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" Nov 01 20:43:25.336169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336124 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" Nov 01 20:43:25.336169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336135 5633 cpu_manager.go:414] "containerMap: removing container" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" Nov 01 20:43:25.336169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336141 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" Nov 01 20:43:25.336169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336155 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" Nov 01 20:43:25.336169 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336162 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336176 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4d936d7e-78b5-4fb4-bb07-0009220fdd78" containerName="revoke-keys" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336182 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d936d7e-78b5-4fb4-bb07-0009220fdd78" containerName="revoke-keys" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336205 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336210 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336223 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b1771e1a-c04c-4257-8cf6-deb67eefa7ac" containerName="payload-processing" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336228 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1771e1a-c04c-4257-8cf6-deb67eefa7ac" containerName="payload-processing" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336237 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336244 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336256 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336261 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336360 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336371 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4deeb6d-3552-4d30-8e78-8e848757a623" containerName="istio-proxy" Nov 01 20:43:25.336372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336380 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5cb13b-c0f3-4b36-b1c2-eaec509c7f45" containerName="main" Nov 01 20:43:25.336808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336386 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" Nov 01 20:43:25.336808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336394 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f1de7c-e045-4fb6-98e4-97972982358c" containerName="main" Nov 01 20:43:25.336808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336409 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d936d7e-78b5-4fb4-bb07-0009220fdd78" containerName="revoke-keys" Nov 01 20:43:25.336808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336419 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7794ca7c-889b-4042-b546-681d2182f466" containerName="istio-proxy" Nov 01 20:43:25.336808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336425 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1771e1a-c04c-4257-8cf6-deb67eefa7ac" containerName="payload-processing" Nov 01 20:43:25.336808 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.336434 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9070b4cc-cc17-4396-9a63-83963140bcc8" containerName="istio-proxy" Nov 01 20:43:25.340263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.340248 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.414395 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.414363 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.414591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.414415 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.414591 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.414549 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnf9j\" (UniqueName: \"kubernetes.io/projected/90ace90c-e17a-4c7f-801c-b497f6aea9e8-kube-api-access-lnf9j\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.441464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.441428 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:43:25.441464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.441457 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:43:25.441464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.441470 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:43:25.441830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.441571 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:43:25.445204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.445178 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf"] Nov 01 20:43:25.516159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.516129 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.516360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.516177 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.516360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.516244 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnf9j\" (UniqueName: \"kubernetes.io/projected/90ace90c-e17a-4c7f-801c-b497f6aea9e8-kube-api-access-lnf9j\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.516902 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.516876 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.518752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.518731 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.572668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.572634 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnf9j\" (UniqueName: \"kubernetes.io/projected/90ace90c-e17a-4c7f-801c-b497f6aea9e8-kube-api-access-lnf9j\") pod \"maas-api-revoke-sub-5f3f541a-b8qnf\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:25.653720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:25.653637 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:26.028498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:26.028475 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf"] Nov 01 20:43:26.029123 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:43:26.029100 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ace90c_e17a_4c7f_801c_b497f6aea9e8.slice/crio-1d4c3bc80926dfec65b7f4493a94fd15ba2a2e3e54d2f803ba8c271f1d12ed4f WatchSource:0}: Error finding container 1d4c3bc80926dfec65b7f4493a94fd15ba2a2e3e54d2f803ba8c271f1d12ed4f: Status 404 returned error can't find the container with id 1d4c3bc80926dfec65b7f4493a94fd15ba2a2e3e54d2f803ba8c271f1d12ed4f Nov 01 20:43:26.244168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:26.244127 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" event={"ID":"90ace90c-e17a-4c7f-801c-b497f6aea9e8","Type":"ContainerStarted","Data":"1d4c3bc80926dfec65b7f4493a94fd15ba2a2e3e54d2f803ba8c271f1d12ed4f"} Nov 01 20:43:27.053459 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:27.053424 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:27.251384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:27.251344 5633 generic.go:356] "Generic (PLEG): container finished" podID="90ace90c-e17a-4c7f-801c-b497f6aea9e8" containerID="e5461d91ee7f7f9dc5a09458313ac3b20b8c1bca344b2a4b56281aefb93755f7" exitCode=0 Nov 01 20:43:27.251792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:27.251420 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" event={"ID":"90ace90c-e17a-4c7f-801c-b497f6aea9e8","Type":"ContainerDied","Data":"e5461d91ee7f7f9dc5a09458313ac3b20b8c1bca344b2a4b56281aefb93755f7"} Nov 01 20:43:28.415252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.415220 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:28.552880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.552795 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-xks-ca\") pod \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " Nov 01 20:43:28.552880 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.552857 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ace90c-e17a-4c7f-801c-b497f6aea9e8-kube-api-access-lnf9j\" (UniqueName: \"kubernetes.io/projected/90ace90c-e17a-4c7f-801c-b497f6aea9e8-kube-api-access-lnf9j\") pod \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " Nov 01 20:43:28.553143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.553044 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-openshift-ca\") pod \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\" (UID: \"90ace90c-e17a-4c7f-801c-b497f6aea9e8\") " Nov 01 20:43:28.553477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.553445 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-openshift-ca" pod "90ace90c-e17a-4c7f-801c-b497f6aea9e8" (UID: "90ace90c-e17a-4c7f-801c-b497f6aea9e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:43:28.555365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.555315 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-xks-ca" pod "90ace90c-e17a-4c7f-801c-b497f6aea9e8" (UID: "90ace90c-e17a-4c7f-801c-b497f6aea9e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:43:28.555505 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.555380 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ace90c-e17a-4c7f-801c-b497f6aea9e8-kube-api-access-lnf9j" pod "90ace90c-e17a-4c7f-801c-b497f6aea9e8" (UID: "90ace90c-e17a-4c7f-801c-b497f6aea9e8"). InnerVolumeSpecName "kube-api-access-lnf9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:43:28.654698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.654662 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:28.654698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.654692 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnf9j\" (UniqueName: \"kubernetes.io/projected/90ace90c-e17a-4c7f-801c-b497f6aea9e8-kube-api-access-lnf9j\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:28.654698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:28.654703 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ace90c-e17a-4c7f-801c-b497f6aea9e8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:29.265508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:29.265475 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" Nov 01 20:43:29.265699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:29.265479 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf" event={"ID":"90ace90c-e17a-4c7f-801c-b497f6aea9e8","Type":"ContainerDied","Data":"1d4c3bc80926dfec65b7f4493a94fd15ba2a2e3e54d2f803ba8c271f1d12ed4f"} Nov 01 20:43:29.265699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:29.265609 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4c3bc80926dfec65b7f4493a94fd15ba2a2e3e54d2f803ba8c271f1d12ed4f" Nov 01 20:43:31.216274 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:31.216240 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:31.403015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:31.402975 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:31.698252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:31.698166 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:36.385281 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:36.385241 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:37.694408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.694371 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9"] Nov 01 20:43:37.694912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.694898 5633 cpu_manager.go:414] "containerMap: removing container" podUID="90ace90c-e17a-4c7f-801c-b497f6aea9e8" containerName="revoke-keys" Nov 01 20:43:37.694958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.694913 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ace90c-e17a-4c7f-801c-b497f6aea9e8" containerName="revoke-keys" Nov 01 20:43:37.695003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.694960 5633 cpu_manager.go:414] "containerMap: removing container" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" Nov 01 20:43:37.695003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.694965 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e7055e-4677-478f-bb8e-4423b79d72c0" containerName="istio-proxy" Nov 01 20:43:37.695074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.695040 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ace90c-e17a-4c7f-801c-b497f6aea9e8" containerName="revoke-keys" Nov 01 20:43:37.698363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.698319 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.750436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.750393 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:43:37.754143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.754110 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9"] Nov 01 20:43:37.798656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.798625 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:43:37.798656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.798633 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:43:37.799871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.799845 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:43:37.863576 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.863538 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.863766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.863726 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.863766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.863757 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6cc4\" (UniqueName: \"kubernetes.io/projected/b5969d01-2f2e-4b0e-958a-8e1f325717cd-kube-api-access-k6cc4\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.965176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.965144 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.965394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.965189 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6cc4\" (UniqueName: \"kubernetes.io/projected/b5969d01-2f2e-4b0e-958a-8e1f325717cd-kube-api-access-k6cc4\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.965394 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.965285 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.965869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.965847 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:37.967857 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:37.967836 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:38.021389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:38.021361 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6cc4\" (UniqueName: \"kubernetes.io/projected/b5969d01-2f2e-4b0e-958a-8e1f325717cd-kube-api-access-k6cc4\") pod \"maas-api-revoke-sub-9cc1a334-mdgw9\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:38.312210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:38.312131 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:38.690525 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:38.690497 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9"] Nov 01 20:43:38.691743 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:43:38.691715 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5969d01_2f2e_4b0e_958a_8e1f325717cd.slice/crio-19ea969feffcc74824bb61084e3ef82608bec5a6778965fd72a11c3fe61544a8 WatchSource:0}: Error finding container 19ea969feffcc74824bb61084e3ef82608bec5a6778965fd72a11c3fe61544a8: Status 404 returned error can't find the container with id 19ea969feffcc74824bb61084e3ef82608bec5a6778965fd72a11c3fe61544a8 Nov 01 20:43:39.329163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:39.329075 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" event={"ID":"b5969d01-2f2e-4b0e-958a-8e1f325717cd","Type":"ContainerStarted","Data":"7c52e269a8c02dca283e28210665e3412d5afa56cb6bd0e8fb2ecf23aba59184"} Nov 01 20:43:39.329163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:39.329151 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" event={"ID":"b5969d01-2f2e-4b0e-958a-8e1f325717cd","Type":"ContainerStarted","Data":"19ea969feffcc74824bb61084e3ef82608bec5a6778965fd72a11c3fe61544a8"} Nov 01 20:43:39.441920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:39.441740 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" podStartSLOduration=2.441717992 podStartE2EDuration="2.441717992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:39.440845732 +0000 UTC m=+2427.115104987" watchObservedRunningTime="2026-11-01 20:43:39.441717992 +0000 UTC m=+2427.115977247" Nov 01 20:43:40.336897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:40.336859 5633 generic.go:356] "Generic (PLEG): container finished" podID="b5969d01-2f2e-4b0e-958a-8e1f325717cd" containerID="7c52e269a8c02dca283e28210665e3412d5afa56cb6bd0e8fb2ecf23aba59184" exitCode=0 Nov 01 20:43:40.337323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:40.336937 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" event={"ID":"b5969d01-2f2e-4b0e-958a-8e1f325717cd","Type":"ContainerDied","Data":"7c52e269a8c02dca283e28210665e3412d5afa56cb6bd0e8fb2ecf23aba59184"} Nov 01 20:43:41.137828 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.137790 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:41.500252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.500224 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:41.604825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.604795 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-openshift-ca\") pod \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " Nov 01 20:43:41.605015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.604867 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5969d01-2f2e-4b0e-958a-8e1f325717cd-kube-api-access-k6cc4\" (UniqueName: \"kubernetes.io/projected/b5969d01-2f2e-4b0e-958a-8e1f325717cd-kube-api-access-k6cc4\") pod \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " Nov 01 20:43:41.605015 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.604907 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-xks-ca\") pod \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\" (UID: \"b5969d01-2f2e-4b0e-958a-8e1f325717cd\") " Nov 01 20:43:41.605210 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.605183 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-openshift-ca" pod "b5969d01-2f2e-4b0e-958a-8e1f325717cd" (UID: "b5969d01-2f2e-4b0e-958a-8e1f325717cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:43:41.605431 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.605402 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:41.607310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.607279 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-xks-ca" pod "b5969d01-2f2e-4b0e-958a-8e1f325717cd" (UID: "b5969d01-2f2e-4b0e-958a-8e1f325717cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:43:41.607310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.607300 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5969d01-2f2e-4b0e-958a-8e1f325717cd-kube-api-access-k6cc4" pod "b5969d01-2f2e-4b0e-958a-8e1f325717cd" (UID: "b5969d01-2f2e-4b0e-958a-8e1f325717cd"). InnerVolumeSpecName "kube-api-access-k6cc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:43:41.707023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.706942 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6cc4\" (UniqueName: \"kubernetes.io/projected/b5969d01-2f2e-4b0e-958a-8e1f325717cd-kube-api-access-k6cc4\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:41.707023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:41.706970 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5969d01-2f2e-4b0e-958a-8e1f325717cd-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:42.351116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:42.351085 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" Nov 01 20:43:42.351116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:42.351108 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9" event={"ID":"b5969d01-2f2e-4b0e-958a-8e1f325717cd","Type":"ContainerDied","Data":"19ea969feffcc74824bb61084e3ef82608bec5a6778965fd72a11c3fe61544a8"} Nov 01 20:43:42.351423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:42.351152 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ea969feffcc74824bb61084e3ef82608bec5a6778965fd72a11c3fe61544a8" Nov 01 20:43:45.793890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:45.793853 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:46.157160 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:43:46.157080 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:43:49.318237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.318198 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh"] Nov 01 20:43:49.318938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.318915 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b5969d01-2f2e-4b0e-958a-8e1f325717cd" containerName="revoke-keys" Nov 01 20:43:49.319056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.318941 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5969d01-2f2e-4b0e-958a-8e1f325717cd" containerName="revoke-keys" Nov 01 20:43:49.319148 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.319134 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5969d01-2f2e-4b0e-958a-8e1f325717cd" containerName="revoke-keys" Nov 01 20:43:49.323287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.323267 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.374745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.374704 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:43:49.376806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.376773 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh"] Nov 01 20:43:49.397096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.397062 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.397288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.397131 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.397288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.397266 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zhd\" (UniqueName: \"kubernetes.io/projected/7f8a8455-bc05-4921-a9ed-b6f1597f613a-kube-api-access-m5zhd\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.421161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.421128 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:43:49.421161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.421146 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:43:49.421161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.421126 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:43:49.498557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.498526 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zhd\" (UniqueName: \"kubernetes.io/projected/7f8a8455-bc05-4921-a9ed-b6f1597f613a-kube-api-access-m5zhd\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.498790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.498671 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.498790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.498734 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.500042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.500006 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.502189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.502160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.555195 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.555160 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zhd\" (UniqueName: \"kubernetes.io/projected/7f8a8455-bc05-4921-a9ed-b6f1597f613a-kube-api-access-m5zhd\") pod \"maas-api-revoke-sub-0be971ce-cm5qh\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:49.638480 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:49.638401 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:50.014722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:50.014694 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh"] Nov 01 20:43:50.015657 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:43:50.015631 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8a8455_bc05_4921_a9ed_b6f1597f613a.slice/crio-bf04471961008e9172b95ee63185d2b67783f92273d4784a32a45f9b9f3513cb WatchSource:0}: Error finding container bf04471961008e9172b95ee63185d2b67783f92273d4784a32a45f9b9f3513cb: Status 404 returned error can't find the container with id bf04471961008e9172b95ee63185d2b67783f92273d4784a32a45f9b9f3513cb Nov 01 20:43:50.409502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:50.409470 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" event={"ID":"7f8a8455-bc05-4921-a9ed-b6f1597f613a","Type":"ContainerStarted","Data":"92338358ceefe10944a8212377a7d8365b9c6fb051c7e2dd0112a14a0e3d2849"} Nov 01 20:43:50.409934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:50.409523 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" event={"ID":"7f8a8455-bc05-4921-a9ed-b6f1597f613a","Type":"ContainerStarted","Data":"bf04471961008e9172b95ee63185d2b67783f92273d4784a32a45f9b9f3513cb"} Nov 01 20:43:50.499285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:50.499253 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:50.522018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:50.521966 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" podStartSLOduration=1.5219508290000001 podStartE2EDuration="1.521950829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:50.520711433 +0000 UTC m=+2438.194970690" watchObservedRunningTime="2026-11-01 20:43:50.521950829 +0000 UTC m=+2438.196210084" Nov 01 20:43:51.416815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:51.416785 5633 generic.go:356] "Generic (PLEG): container finished" podID="7f8a8455-bc05-4921-a9ed-b6f1597f613a" containerID="92338358ceefe10944a8212377a7d8365b9c6fb051c7e2dd0112a14a0e3d2849" exitCode=0 Nov 01 20:43:51.417227 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:51.416881 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" event={"ID":"7f8a8455-bc05-4921-a9ed-b6f1597f613a","Type":"ContainerDied","Data":"92338358ceefe10944a8212377a7d8365b9c6fb051c7e2dd0112a14a0e3d2849"} Nov 01 20:43:52.577365 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.577307 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:52.636310 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.636280 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-xks-ca\") pod \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " Nov 01 20:43:52.636538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.636322 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8a8455-bc05-4921-a9ed-b6f1597f613a-kube-api-access-m5zhd\" (UniqueName: \"kubernetes.io/projected/7f8a8455-bc05-4921-a9ed-b6f1597f613a-kube-api-access-m5zhd\") pod \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " Nov 01 20:43:52.636538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.636438 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-openshift-ca\") pod \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\" (UID: \"7f8a8455-bc05-4921-a9ed-b6f1597f613a\") " Nov 01 20:43:52.636847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.636810 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-openshift-ca" pod "7f8a8455-bc05-4921-a9ed-b6f1597f613a" (UID: "7f8a8455-bc05-4921-a9ed-b6f1597f613a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:43:52.636987 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.636972 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:52.638669 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.638644 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8a8455-bc05-4921-a9ed-b6f1597f613a-kube-api-access-m5zhd" pod "7f8a8455-bc05-4921-a9ed-b6f1597f613a" (UID: "7f8a8455-bc05-4921-a9ed-b6f1597f613a"). InnerVolumeSpecName "kube-api-access-m5zhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:43:52.638760 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.638665 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-xks-ca" pod "7f8a8455-bc05-4921-a9ed-b6f1597f613a" (UID: "7f8a8455-bc05-4921-a9ed-b6f1597f613a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:43:52.737778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.737737 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f8a8455-bc05-4921-a9ed-b6f1597f613a-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:52.737778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:52.737771 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5zhd\" (UniqueName: \"kubernetes.io/projected/7f8a8455-bc05-4921-a9ed-b6f1597f613a-kube-api-access-m5zhd\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:43:53.430704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:53.430673 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" Nov 01 20:43:53.430704 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:53.430696 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh" event={"ID":"7f8a8455-bc05-4921-a9ed-b6f1597f613a","Type":"ContainerDied","Data":"bf04471961008e9172b95ee63185d2b67783f92273d4784a32a45f9b9f3513cb"} Nov 01 20:43:53.430933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:53.430747 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf04471961008e9172b95ee63185d2b67783f92273d4784a32a45f9b9f3513cb" Nov 01 20:43:55.171011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:55.170963 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:57.055253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:57.055217 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:43:59.830424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:43:59.830387 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:02.827657 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:02.827620 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:03.001564 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:03.001524 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:04.514345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.514297 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5"] Nov 01 20:44:04.514888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.514873 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7f8a8455-bc05-4921-a9ed-b6f1597f613a" containerName="revoke-keys" Nov 01 20:44:04.514942 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.514889 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8a8455-bc05-4921-a9ed-b6f1597f613a" containerName="revoke-keys" Nov 01 20:44:04.514990 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.514981 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8a8455-bc05-4921-a9ed-b6f1597f613a" containerName="revoke-keys" Nov 01 20:44:04.518599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.518581 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.539010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.538967 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:04.570516 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.570479 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:44:04.570724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.570554 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:44:04.570724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.570561 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:44:04.619986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.619955 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:44:04.623729 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.623697 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5"] Nov 01 20:44:04.678052 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.678016 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2xs6\" (UniqueName: \"kubernetes.io/projected/e8291468-b8de-4e33-9456-5dc06ab4f216-kube-api-access-m2xs6\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.678231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.678089 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.678231 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.678163 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.779913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.779833 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2xs6\" (UniqueName: \"kubernetes.io/projected/e8291468-b8de-4e33-9456-5dc06ab4f216-kube-api-access-m2xs6\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.780082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.779910 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.780082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.779937 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.780713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.780688 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.782502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.782483 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:04.836067 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:04.836037 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2xs6\" (UniqueName: \"kubernetes.io/projected/e8291468-b8de-4e33-9456-5dc06ab4f216-kube-api-access-m2xs6\") pod \"maas-api-revoke-sub-33c3eb4b-5pxm5\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:05.132498 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:05.132416 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:05.507270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:05.507238 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5"] Nov 01 20:44:05.507937 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:44:05.507907 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8291468_b8de_4e33_9456_5dc06ab4f216.slice/crio-8a26af7eef7ffa4ac44bc5a9a0ee0cbdc5fe5fb20c30e83c41e9278a96c22d4f WatchSource:0}: Error finding container 8a26af7eef7ffa4ac44bc5a9a0ee0cbdc5fe5fb20c30e83c41e9278a96c22d4f: Status 404 returned error can't find the container with id 8a26af7eef7ffa4ac44bc5a9a0ee0cbdc5fe5fb20c30e83c41e9278a96c22d4f Nov 01 20:44:05.516306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:05.516273 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" event={"ID":"e8291468-b8de-4e33-9456-5dc06ab4f216","Type":"ContainerStarted","Data":"8a26af7eef7ffa4ac44bc5a9a0ee0cbdc5fe5fb20c30e83c41e9278a96c22d4f"} Nov 01 20:44:05.973849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:05.973812 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:44:06.524521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:06.524486 5633 generic.go:356] "Generic (PLEG): container finished" podID="e8291468-b8de-4e33-9456-5dc06ab4f216" containerID="bcb0698056b2932b6400d6027adff82b1ff12fe1462bfa3c6c0526b8e21b2e71" exitCode=0 Nov 01 20:44:06.524983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:06.524614 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" event={"ID":"e8291468-b8de-4e33-9456-5dc06ab4f216","Type":"ContainerDied","Data":"bcb0698056b2932b6400d6027adff82b1ff12fe1462bfa3c6c0526b8e21b2e71"} Nov 01 20:44:07.681899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.681874 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:07.713140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.713109 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8291468-b8de-4e33-9456-5dc06ab4f216-kube-api-access-m2xs6\" (UniqueName: \"kubernetes.io/projected/e8291468-b8de-4e33-9456-5dc06ab4f216-kube-api-access-m2xs6\") pod \"e8291468-b8de-4e33-9456-5dc06ab4f216\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " Nov 01 20:44:07.713417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.713241 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-openshift-ca\") pod \"e8291468-b8de-4e33-9456-5dc06ab4f216\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " Nov 01 20:44:07.713417 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.713365 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-xks-ca\") pod \"e8291468-b8de-4e33-9456-5dc06ab4f216\" (UID: \"e8291468-b8de-4e33-9456-5dc06ab4f216\") " Nov 01 20:44:07.713658 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.713635 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-openshift-ca" pod "e8291468-b8de-4e33-9456-5dc06ab4f216" (UID: "e8291468-b8de-4e33-9456-5dc06ab4f216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:07.715474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.715442 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8291468-b8de-4e33-9456-5dc06ab4f216-kube-api-access-m2xs6" pod "e8291468-b8de-4e33-9456-5dc06ab4f216" (UID: "e8291468-b8de-4e33-9456-5dc06ab4f216"). InnerVolumeSpecName "kube-api-access-m2xs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:07.715474 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.715461 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-xks-ca" pod "e8291468-b8de-4e33-9456-5dc06ab4f216" (UID: "e8291468-b8de-4e33-9456-5dc06ab4f216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:44:07.814786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.814715 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:07.814786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.814743 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8291468-b8de-4e33-9456-5dc06ab4f216-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:07.814786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:07.814752 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2xs6\" (UniqueName: \"kubernetes.io/projected/e8291468-b8de-4e33-9456-5dc06ab4f216-kube-api-access-m2xs6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:08.539842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:08.539814 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" Nov 01 20:44:08.539842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:08.539832 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5" event={"ID":"e8291468-b8de-4e33-9456-5dc06ab4f216","Type":"ContainerDied","Data":"8a26af7eef7ffa4ac44bc5a9a0ee0cbdc5fe5fb20c30e83c41e9278a96c22d4f"} Nov 01 20:44:08.540111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:08.539890 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a26af7eef7ffa4ac44bc5a9a0ee0cbdc5fe5fb20c30e83c41e9278a96c22d4f" Nov 01 20:44:09.231163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:09.231126 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:13.892756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:13.892722 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:16.156226 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:44:16.156191 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:44:16.763650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:16.763611 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:16.892830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:16.892782 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:17.582033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.581989 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn"] Nov 01 20:44:17.582562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.582548 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e8291468-b8de-4e33-9456-5dc06ab4f216" containerName="revoke-keys" Nov 01 20:44:17.582610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.582563 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8291468-b8de-4e33-9456-5dc06ab4f216" containerName="revoke-keys" Nov 01 20:44:17.582662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.582653 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8291468-b8de-4e33-9456-5dc06ab4f216" containerName="revoke-keys" Nov 01 20:44:17.586355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.586315 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.635140 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.635109 5633 scope.go:122] "RemoveContainer" containerID="ecd0be992badf00e36d1ac55ad310ed7914a069bf6f52ab7c0a53e095e33e169" Nov 01 20:44:17.637805 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.637780 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31038fb4-07b6-413c-b139-05ca83384495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.638103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.638070 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghvz\" (UniqueName: \"kubernetes.io/projected/31038fb4-07b6-413c-b139-05ca83384495-kube-api-access-fghvz\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.638342 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.638242 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31038fb4-07b6-413c-b139-05ca83384495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.643016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.642985 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn"] Nov 01 20:44:17.648201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.648172 5633 scope.go:122] "RemoveContainer" containerID="d2f0cd41760382faa5449ca79ae2e92797c58bf9133b099f217a645088a5e74c" Nov 01 20:44:17.687798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.687766 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:44:17.687933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.687807 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:44:17.687994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.687939 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:44:17.687994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.687962 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:44:17.739910 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.739877 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31038fb4-07b6-413c-b139-05ca83384495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.740119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.739990 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghvz\" (UniqueName: \"kubernetes.io/projected/31038fb4-07b6-413c-b139-05ca83384495-kube-api-access-fghvz\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.740119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.740034 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31038fb4-07b6-413c-b139-05ca83384495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.740730 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.740704 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31038fb4-07b6-413c-b139-05ca83384495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.742411 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.742382 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31038fb4-07b6-413c-b139-05ca83384495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.801801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.801764 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghvz\" (UniqueName: \"kubernetes.io/projected/31038fb4-07b6-413c-b139-05ca83384495-kube-api-access-fghvz\") pod \"maas-api-revoke-sub-3234497e-bjxkn\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:17.902912 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:17.902824 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:18.305006 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:18.302580 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn"] Nov 01 20:44:18.308993 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:44:18.308946 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31038fb4_07b6_413c_b139_05ca83384495.slice/crio-0fd3525e4355e08bce72e535ce05da12f3a5cd53515639f07a64821b113fee2a WatchSource:0}: Error finding container 0fd3525e4355e08bce72e535ce05da12f3a5cd53515639f07a64821b113fee2a: Status 404 returned error can't find the container with id 0fd3525e4355e08bce72e535ce05da12f3a5cd53515639f07a64821b113fee2a Nov 01 20:44:18.606508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:18.606479 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" event={"ID":"31038fb4-07b6-413c-b139-05ca83384495","Type":"ContainerStarted","Data":"0fd3525e4355e08bce72e535ce05da12f3a5cd53515639f07a64821b113fee2a"} Nov 01 20:44:18.620924 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:18.620890 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:19.613972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:19.613934 5633 generic.go:356] "Generic (PLEG): container finished" podID="31038fb4-07b6-413c-b139-05ca83384495" containerID="5e9609e9c80e2aec4fa857495b0c3649d4a8ec261a3340a600eb90c48c55f692" exitCode=0 Nov 01 20:44:19.613972 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:19.613977 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" event={"ID":"31038fb4-07b6-413c-b139-05ca83384495","Type":"ContainerDied","Data":"5e9609e9c80e2aec4fa857495b0c3649d4a8ec261a3340a600eb90c48c55f692"} Nov 01 20:44:20.179984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.179930 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:44:20.765827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.765787 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:20.877205 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.877117 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31038fb4-07b6-413c-b139-05ca83384495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31038fb4-07b6-413c-b139-05ca83384495-maas-api-xks-ca\") pod \"31038fb4-07b6-413c-b139-05ca83384495\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " Nov 01 20:44:20.877495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.877238 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31038fb4-07b6-413c-b139-05ca83384495-kube-api-access-fghvz\" (UniqueName: \"kubernetes.io/projected/31038fb4-07b6-413c-b139-05ca83384495-kube-api-access-fghvz\") pod \"31038fb4-07b6-413c-b139-05ca83384495\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " Nov 01 20:44:20.877495 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.877310 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31038fb4-07b6-413c-b139-05ca83384495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31038fb4-07b6-413c-b139-05ca83384495-maas-api-openshift-ca\") pod \"31038fb4-07b6-413c-b139-05ca83384495\" (UID: \"31038fb4-07b6-413c-b139-05ca83384495\") " Nov 01 20:44:20.878389 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.878357 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31038fb4-07b6-413c-b139-05ca83384495-maas-api-openshift-ca" pod "31038fb4-07b6-413c-b139-05ca83384495" (UID: "31038fb4-07b6-413c-b139-05ca83384495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:20.880946 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.880921 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31038fb4-07b6-413c-b139-05ca83384495-maas-api-xks-ca" pod "31038fb4-07b6-413c-b139-05ca83384495" (UID: "31038fb4-07b6-413c-b139-05ca83384495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:44:20.881139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.881111 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31038fb4-07b6-413c-b139-05ca83384495-kube-api-access-fghvz" pod "31038fb4-07b6-413c-b139-05ca83384495" (UID: "31038fb4-07b6-413c-b139-05ca83384495"). InnerVolumeSpecName "kube-api-access-fghvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:20.979038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.979003 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghvz\" (UniqueName: \"kubernetes.io/projected/31038fb4-07b6-413c-b139-05ca83384495-kube-api-access-fghvz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:20.979038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.979033 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31038fb4-07b6-413c-b139-05ca83384495-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:20.979038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:20.979043 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31038fb4-07b6-413c-b139-05ca83384495-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:21.628789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:21.628745 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" Nov 01 20:44:21.628789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:21.628765 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn" event={"ID":"31038fb4-07b6-413c-b139-05ca83384495","Type":"ContainerDied","Data":"0fd3525e4355e08bce72e535ce05da12f3a5cd53515639f07a64821b113fee2a"} Nov 01 20:44:21.629021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:21.628823 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd3525e4355e08bce72e535ce05da12f3a5cd53515639f07a64821b113fee2a" Nov 01 20:44:23.285366 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:23.285294 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:26.029714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:26.029655 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:26.199656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:26.199624 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:27.986209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:27.986168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:29.692364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:29.692295 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:29.819155 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:29.819117 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:30.475493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.471392 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw"] Nov 01 20:44:30.475493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.472640 5633 cpu_manager.go:414] "containerMap: removing container" podUID="31038fb4-07b6-413c-b139-05ca83384495" containerName="revoke-keys" Nov 01 20:44:30.475493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.472662 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31038fb4-07b6-413c-b139-05ca83384495" containerName="revoke-keys" Nov 01 20:44:30.475493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.472951 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="31038fb4-07b6-413c-b139-05ca83384495" containerName="revoke-keys" Nov 01 20:44:30.477895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.477862 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.528306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.528273 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw"] Nov 01 20:44:30.572734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.572697 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:44:30.572734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.572708 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:44:30.572734 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.572711 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:44:30.572999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.572746 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:44:30.591400 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.591366 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.591566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.591540 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljsg\" (UniqueName: \"kubernetes.io/projected/60e22317-6474-4ab7-87bf-e7b020abe7e9-kube-api-access-nljsg\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.591608 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.591583 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.693078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.693039 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.693583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.693148 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljsg\" (UniqueName: \"kubernetes.io/projected/60e22317-6474-4ab7-87bf-e7b020abe7e9-kube-api-access-nljsg\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.693583 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.693198 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.693870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.693851 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.695785 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.695762 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.750271 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.750189 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljsg\" (UniqueName: \"kubernetes.io/projected/60e22317-6474-4ab7-87bf-e7b020abe7e9-kube-api-access-nljsg\") pod \"maas-api-revoke-sub-3933d258-6vptw\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:30.791981 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:30.791944 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:31.169438 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:31.169404 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw"] Nov 01 20:44:31.169779 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:44:31.169743 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e22317_6474_4ab7_87bf_e7b020abe7e9.slice/crio-9388fd7101696bbadc881a87452ddead9bd95b76b4b62e767c45ee99ae3174a7 WatchSource:0}: Error finding container 9388fd7101696bbadc881a87452ddead9bd95b76b4b62e767c45ee99ae3174a7: Status 404 returned error can't find the container with id 9388fd7101696bbadc881a87452ddead9bd95b76b4b62e767c45ee99ae3174a7 Nov 01 20:44:31.555179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:31.555141 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:31.695092 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:31.695048 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" event={"ID":"60e22317-6474-4ab7-87bf-e7b020abe7e9","Type":"ContainerStarted","Data":"0aca7c6998df22d268c5dc8e8e069231b0c7b2d36f9bf1a10f653196878a1660"} Nov 01 20:44:31.695539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:31.695113 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" event={"ID":"60e22317-6474-4ab7-87bf-e7b020abe7e9","Type":"ContainerStarted","Data":"9388fd7101696bbadc881a87452ddead9bd95b76b4b62e767c45ee99ae3174a7"} Nov 01 20:44:31.809766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:31.809658 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" podStartSLOduration=1.809640838 podStartE2EDuration="1.809640838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:31.807276989 +0000 UTC m=+2479.481536244" watchObservedRunningTime="2026-11-01 20:44:31.809640838 +0000 UTC m=+2479.483900093" Nov 01 20:44:32.695598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:32.695555 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:32.703293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:32.703259 5633 generic.go:356] "Generic (PLEG): container finished" podID="60e22317-6474-4ab7-87bf-e7b020abe7e9" containerID="0aca7c6998df22d268c5dc8e8e069231b0c7b2d36f9bf1a10f653196878a1660" exitCode=0 Nov 01 20:44:32.703293 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:32.703355 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" event={"ID":"60e22317-6474-4ab7-87bf-e7b020abe7e9","Type":"ContainerDied","Data":"0aca7c6998df22d268c5dc8e8e069231b0c7b2d36f9bf1a10f653196878a1660"} Nov 01 20:44:33.873810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.873775 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:33.926220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.926184 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-xks-ca\") pod \"60e22317-6474-4ab7-87bf-e7b020abe7e9\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " Nov 01 20:44:33.926432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.926304 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e22317-6474-4ab7-87bf-e7b020abe7e9-kube-api-access-nljsg\" (UniqueName: \"kubernetes.io/projected/60e22317-6474-4ab7-87bf-e7b020abe7e9-kube-api-access-nljsg\") pod \"60e22317-6474-4ab7-87bf-e7b020abe7e9\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " Nov 01 20:44:33.926432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.926326 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-openshift-ca\") pod \"60e22317-6474-4ab7-87bf-e7b020abe7e9\" (UID: \"60e22317-6474-4ab7-87bf-e7b020abe7e9\") " Nov 01 20:44:33.926705 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.926680 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-openshift-ca" pod "60e22317-6474-4ab7-87bf-e7b020abe7e9" (UID: "60e22317-6474-4ab7-87bf-e7b020abe7e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:33.926878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.926862 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:33.928549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.928529 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-xks-ca" pod "60e22317-6474-4ab7-87bf-e7b020abe7e9" (UID: "60e22317-6474-4ab7-87bf-e7b020abe7e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:44:33.928610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:33.928546 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e22317-6474-4ab7-87bf-e7b020abe7e9-kube-api-access-nljsg" pod "60e22317-6474-4ab7-87bf-e7b020abe7e9" (UID: "60e22317-6474-4ab7-87bf-e7b020abe7e9"). InnerVolumeSpecName "kube-api-access-nljsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:34.028091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:34.028052 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e22317-6474-4ab7-87bf-e7b020abe7e9-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:34.028091 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:34.028095 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljsg\" (UniqueName: \"kubernetes.io/projected/60e22317-6474-4ab7-87bf-e7b020abe7e9-kube-api-access-nljsg\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:34.718244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:34.718212 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" Nov 01 20:44:34.718244 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:34.718234 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw" event={"ID":"60e22317-6474-4ab7-87bf-e7b020abe7e9","Type":"ContainerDied","Data":"9388fd7101696bbadc881a87452ddead9bd95b76b4b62e767c45ee99ae3174a7"} Nov 01 20:44:34.718490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:34.718316 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9388fd7101696bbadc881a87452ddead9bd95b76b4b62e767c45ee99ae3174a7" Nov 01 20:44:34.984005 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:34.983971 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:44:37.408908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.408871 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:37.940284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.940250 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2"] Nov 01 20:44:37.940863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.940848 5633 cpu_manager.go:414] "containerMap: removing container" podUID="60e22317-6474-4ab7-87bf-e7b020abe7e9" containerName="revoke-keys" Nov 01 20:44:37.940915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.940864 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e22317-6474-4ab7-87bf-e7b020abe7e9" containerName="revoke-keys" Nov 01 20:44:37.940965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.940957 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e22317-6474-4ab7-87bf-e7b020abe7e9" containerName="revoke-keys" Nov 01 20:44:37.944049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.944034 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:37.982685 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.982647 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt52g\" (UniqueName: \"kubernetes.io/projected/a1304efc-7d24-46d1-a2ca-c741105f7ff9-kube-api-access-xt52g\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:37.982859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.982699 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:37.982913 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:37.982899 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.000112 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.000069 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2"] Nov 01 20:44:38.044688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.044657 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:44:38.044886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.044733 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:44:38.083897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.083859 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt52g\" (UniqueName: \"kubernetes.io/projected/a1304efc-7d24-46d1-a2ca-c741105f7ff9-kube-api-access-xt52g\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.084103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.083907 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.084103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.084017 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.086630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.086604 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.094201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.094177 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:44:38.094322 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.094218 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:44:38.104901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.104871 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.140353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.140306 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt52g\" (UniqueName: \"kubernetes.io/projected/a1304efc-7d24-46d1-a2ca-c741105f7ff9-kube-api-access-xt52g\") pod \"maas-api-revoke-sub-21400cc4-fc7b2\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.258044 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.258007 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:38.639491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.639459 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2"] Nov 01 20:44:38.640124 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:44:38.640097 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1304efc_7d24_46d1_a2ca_c741105f7ff9.slice/crio-34b89f0b8a5921366ae249e5b1d31b3a18073d0d4dbfd8ab41ecdb25328a877b WatchSource:0}: Error finding container 34b89f0b8a5921366ae249e5b1d31b3a18073d0d4dbfd8ab41ecdb25328a877b: Status 404 returned error can't find the container with id 34b89f0b8a5921366ae249e5b1d31b3a18073d0d4dbfd8ab41ecdb25328a877b Nov 01 20:44:38.746054 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:38.746012 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" event={"ID":"a1304efc-7d24-46d1-a2ca-c741105f7ff9","Type":"ContainerStarted","Data":"34b89f0b8a5921366ae249e5b1d31b3a18073d0d4dbfd8ab41ecdb25328a877b"} Nov 01 20:44:39.395863 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:39.395825 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:39.580871 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:39.580825 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:39.754668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:39.754627 5633 generic.go:356] "Generic (PLEG): container finished" podID="a1304efc-7d24-46d1-a2ca-c741105f7ff9" containerID="45579878cba531449c08d6eb55671ebe411ab8793e8ab7c28f5a69499ccced99" exitCode=0 Nov 01 20:44:39.755078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:39.754706 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" event={"ID":"a1304efc-7d24-46d1-a2ca-c741105f7ff9","Type":"ContainerDied","Data":"45579878cba531449c08d6eb55671ebe411ab8793e8ab7c28f5a69499ccced99"} Nov 01 20:44:40.918565 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:40.918534 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:41.017513 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.017419 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-xks-ca\") pod \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " Nov 01 20:44:41.017698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.017558 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1304efc-7d24-46d1-a2ca-c741105f7ff9-kube-api-access-xt52g\" (UniqueName: \"kubernetes.io/projected/a1304efc-7d24-46d1-a2ca-c741105f7ff9-kube-api-access-xt52g\") pod \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " Nov 01 20:44:41.017698 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.017660 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-openshift-ca\") pod \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\" (UID: \"a1304efc-7d24-46d1-a2ca-c741105f7ff9\") " Nov 01 20:44:41.018078 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.018049 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-openshift-ca" pod "a1304efc-7d24-46d1-a2ca-c741105f7ff9" (UID: "a1304efc-7d24-46d1-a2ca-c741105f7ff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:41.019769 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.019747 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-xks-ca" pod "a1304efc-7d24-46d1-a2ca-c741105f7ff9" (UID: "a1304efc-7d24-46d1-a2ca-c741105f7ff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:44:41.019844 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.019827 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1304efc-7d24-46d1-a2ca-c741105f7ff9-kube-api-access-xt52g" pod "a1304efc-7d24-46d1-a2ca-c741105f7ff9" (UID: "a1304efc-7d24-46d1-a2ca-c741105f7ff9"). InnerVolumeSpecName "kube-api-access-xt52g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:41.119267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.119222 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:41.119267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.119262 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1304efc-7d24-46d1-a2ca-c741105f7ff9-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:41.119267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.119277 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt52g\" (UniqueName: \"kubernetes.io/projected/a1304efc-7d24-46d1-a2ca-c741105f7ff9-kube-api-access-xt52g\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:41.769299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.769270 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" Nov 01 20:44:41.769299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.769278 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2" event={"ID":"a1304efc-7d24-46d1-a2ca-c741105f7ff9","Type":"ContainerDied","Data":"34b89f0b8a5921366ae249e5b1d31b3a18073d0d4dbfd8ab41ecdb25328a877b"} Nov 01 20:44:41.769527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:41.769321 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34b89f0b8a5921366ae249e5b1d31b3a18073d0d4dbfd8ab41ecdb25328a877b" Nov 01 20:44:42.086811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:42.086721 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:42.785893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:42.785859 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:42.914778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:42.914723 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:43.526958 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:43.526918 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:44.046537 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:44.046499 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg"] Nov 01 20:44:44.094530 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:44.094491 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8678da75-xzbxg"] Nov 01 20:44:44.468138 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:44.468109 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbea7954-e0d1-476c-b3d4-48262b4c8ff8" path="/var/lib/kubelet/pods/cbea7954-e0d1-476c-b3d4-48262b4c8ff8/volumes" Nov 01 20:44:45.437660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:45.437621 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:46.158140 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:44:46.158093 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:44:46.895403 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:46.895325 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:47.129925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:47.129881 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:44:49.483699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:49.483650 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:49.677856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:49.677822 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:50.320377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:50.320304 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:50.489827 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:50.489789 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:51.669266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.669227 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:51.727766 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.727722 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh"] Nov 01 20:44:51.728693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.728661 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a1304efc-7d24-46d1-a2ca-c741105f7ff9" containerName="revoke-keys" Nov 01 20:44:51.728693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.728691 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1304efc-7d24-46d1-a2ca-c741105f7ff9" containerName="revoke-keys" Nov 01 20:44:51.729076 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.728881 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1304efc-7d24-46d1-a2ca-c741105f7ff9" containerName="revoke-keys" Nov 01 20:44:51.734288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.734265 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.772294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.772258 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x246l\" (UniqueName: \"kubernetes.io/projected/56286159-4951-47f7-9ed2-3aafde0b09de-kube-api-access-x246l\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.772521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.772474 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.772592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.772524 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.785228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.785192 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh"] Nov 01 20:44:51.829881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.829848 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:44:51.829881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.829867 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:44:51.829881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.829867 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:44:51.836214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.836190 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:44:51.874068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.874031 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.874239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.874084 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.874239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.874172 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x246l\" (UniqueName: \"kubernetes.io/projected/56286159-4951-47f7-9ed2-3aafde0b09de-kube-api-access-x246l\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.874883 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.874805 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.876638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.876618 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:51.930584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:51.930509 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x246l\" (UniqueName: \"kubernetes.io/projected/56286159-4951-47f7-9ed2-3aafde0b09de-kube-api-access-x246l\") pod \"maas-api-revoke-sub-dd5c92d6-mnpmh\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:52.055177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:52.055145 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:52.430751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:52.430723 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh"] Nov 01 20:44:52.431292 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:44:52.431272 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56286159_4951_47f7_9ed2_3aafde0b09de.slice/crio-4f695d7196f0b41de47210b44c5b711e58666bd736818ad1adfb85434d293b27 WatchSource:0}: Error finding container 4f695d7196f0b41de47210b44c5b711e58666bd736818ad1adfb85434d293b27: Status 404 returned error can't find the container with id 4f695d7196f0b41de47210b44c5b711e58666bd736818ad1adfb85434d293b27 Nov 01 20:44:52.860493 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:52.860457 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" event={"ID":"56286159-4951-47f7-9ed2-3aafde0b09de","Type":"ContainerStarted","Data":"6a9146cdbcf84afcc7508a0593ce65888d3ede2686204a238cd48c7132553040"} Nov 01 20:44:52.860985 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:52.860511 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" event={"ID":"56286159-4951-47f7-9ed2-3aafde0b09de","Type":"ContainerStarted","Data":"4f695d7196f0b41de47210b44c5b711e58666bd736818ad1adfb85434d293b27"} Nov 01 20:44:53.886038 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:53.885990 5633 generic.go:356] "Generic (PLEG): container finished" podID="56286159-4951-47f7-9ed2-3aafde0b09de" containerID="6a9146cdbcf84afcc7508a0593ce65888d3ede2686204a238cd48c7132553040" exitCode=0 Nov 01 20:44:53.886919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:53.886887 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" event={"ID":"56286159-4951-47f7-9ed2-3aafde0b09de","Type":"ContainerDied","Data":"6a9146cdbcf84afcc7508a0593ce65888d3ede2686204a238cd48c7132553040"} Nov 01 20:44:55.733876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:55.730429 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:56.129360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.129057 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:56.258697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.258591 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:56.344357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.343630 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-openshift-ca\") pod \"56286159-4951-47f7-9ed2-3aafde0b09de\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " Nov 01 20:44:56.351357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.348593 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-openshift-ca" pod "56286159-4951-47f7-9ed2-3aafde0b09de" (UID: "56286159-4951-47f7-9ed2-3aafde0b09de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:56.351357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.348830 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56286159-4951-47f7-9ed2-3aafde0b09de-kube-api-access-x246l\" (UniqueName: \"kubernetes.io/projected/56286159-4951-47f7-9ed2-3aafde0b09de-kube-api-access-x246l\") pod \"56286159-4951-47f7-9ed2-3aafde0b09de\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " Nov 01 20:44:56.351357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.348978 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-xks-ca\") pod \"56286159-4951-47f7-9ed2-3aafde0b09de\" (UID: \"56286159-4951-47f7-9ed2-3aafde0b09de\") " Nov 01 20:44:56.351357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.349528 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:56.370444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.368131 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56286159-4951-47f7-9ed2-3aafde0b09de-kube-api-access-x246l" pod "56286159-4951-47f7-9ed2-3aafde0b09de" (UID: "56286159-4951-47f7-9ed2-3aafde0b09de"). InnerVolumeSpecName "kube-api-access-x246l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:56.389978 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.389937 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-xks-ca" pod "56286159-4951-47f7-9ed2-3aafde0b09de" (UID: "56286159-4951-47f7-9ed2-3aafde0b09de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:44:56.565423 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.563569 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:56.621849 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.621814 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x246l\" (UniqueName: \"kubernetes.io/projected/56286159-4951-47f7-9ed2-3aafde0b09de-kube-api-access-x246l\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:56.622068 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.622050 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56286159-4951-47f7-9ed2-3aafde0b09de-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:56.979457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:56.979371 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:57.056159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:57.056110 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr"] Nov 01 20:44:57.066139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:57.066071 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c0a576-hr7gr"] Nov 01 20:44:57.113288 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:57.113191 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:44:57.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:57.117742 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" Nov 01 20:44:57.117950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:57.117773 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh" event={"ID":"56286159-4951-47f7-9ed2-3aafde0b09de","Type":"ContainerDied","Data":"4f695d7196f0b41de47210b44c5b711e58666bd736818ad1adfb85434d293b27"} Nov 01 20:44:57.117950 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:57.117825 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f695d7196f0b41de47210b44c5b711e58666bd736818ad1adfb85434d293b27" Nov 01 20:44:58.467687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:58.467651 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d936d7e-78b5-4fb4-bb07-0009220fdd78" path="/var/lib/kubelet/pods/4d936d7e-78b5-4fb4-bb07-0009220fdd78/volumes" Nov 01 20:44:59.523380 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:44:59.523324 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-401f7dd59863: secret "agc-cred-401f7dd59863" not found Nov 01 20:44:59.523380 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:44:59.523378 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-63b1876e63ce: secret "agc-cred-63b1876e63ce" not found Nov 01 20:44:59.523883 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:44:59.523391 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9: [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:44:59.523883 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:44:59.523463 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials podName:47e4a7e2-cfd5-4459-8714-edf33451e2ac nodeName:}" failed. No retries permitted until 2026-11-01 20:45:00.02344214 +0000 UTC m=+2507.697701387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials") pod "payload-processing-external-model-6dcf58fd8-8dvn9" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac") : [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:44:59.922710 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:44:59.922625 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:00.030647 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:00.030604 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-401f7dd59863: secret "agc-cred-401f7dd59863" not found Nov 01 20:45:00.030647 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:00.030636 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-63b1876e63ce: secret "agc-cred-63b1876e63ce" not found Nov 01 20:45:00.030647 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:00.030646 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9: [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:00.030957 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:00.030705 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials podName:47e4a7e2-cfd5-4459-8714-edf33451e2ac nodeName:}" failed. No retries permitted until 2026-11-01 20:45:01.030691644 +0000 UTC m=+2508.704950890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials") pod "payload-processing-external-model-6dcf58fd8-8dvn9" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac") : [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:00.157611 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.157574 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q"] Nov 01 20:45:00.158428 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.158409 5633 cpu_manager.go:414] "containerMap: removing container" podUID="56286159-4951-47f7-9ed2-3aafde0b09de" containerName="revoke-keys" Nov 01 20:45:00.158485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.158431 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56286159-4951-47f7-9ed2-3aafde0b09de" containerName="revoke-keys" Nov 01 20:45:00.158595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.158584 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="56286159-4951-47f7-9ed2-3aafde0b09de" containerName="revoke-keys" Nov 01 20:45:00.163920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.163899 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n"] Nov 01 20:45:00.164037 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.164022 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.168569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.168547 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2"] Nov 01 20:45:00.168697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.168641 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.172240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.172220 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.215944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.215903 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:45:00.215944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.215928 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:45:00.215944 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.215937 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:45:00.233178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.233149 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88479da-823a-482b-8fb1-cb680c329d05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.233372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.233201 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a00de5-a2dd-4a89-a10d-57993d24f88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-jr47n\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.233372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.233227 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6nj8\" (UniqueName: \"kubernetes.io/projected/a88479da-823a-482b-8fb1-cb680c329d05-kube-api-access-n6nj8\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.233465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.233423 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnbk\" (UniqueName: \"kubernetes.io/projected/89a00de5-a2dd-4a89-a10d-57993d24f88f-kube-api-access-sfnbk\") pod \"maas-api-key-cleanup-29892765-jr47n\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.233519 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.233474 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlgr\" (UniqueName: \"kubernetes.io/projected/53cb4a4a-12c5-48b3-89af-85ce2713e530-kube-api-access-9jlgr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.233560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.233549 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4a4a-12c5-48b3-89af-85ce2713e530-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.270133 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.270092 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q"] Nov 01 20:45:00.276173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.276142 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2"] Nov 01 20:45:00.282202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.282169 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n"] Nov 01 20:45:00.334859 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.334826 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnbk\" (UniqueName: \"kubernetes.io/projected/89a00de5-a2dd-4a89-a10d-57993d24f88f-kube-api-access-sfnbk\") pod \"maas-api-key-cleanup-29892765-jr47n\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.335063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.334873 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlgr\" (UniqueName: \"kubernetes.io/projected/53cb4a4a-12c5-48b3-89af-85ce2713e530-kube-api-access-9jlgr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.335063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.334910 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4a4a-12c5-48b3-89af-85ce2713e530-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.335063 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.335039 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88479da-823a-482b-8fb1-cb680c329d05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.335243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.335076 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a00de5-a2dd-4a89-a10d-57993d24f88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-jr47n\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.335243 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.335105 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6nj8\" (UniqueName: \"kubernetes.io/projected/a88479da-823a-482b-8fb1-cb680c329d05-kube-api-access-n6nj8\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.335789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.335763 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88479da-823a-482b-8fb1-cb680c329d05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.335789 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.335780 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a00de5-a2dd-4a89-a10d-57993d24f88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-jr47n\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.335965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.335779 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4a4a-12c5-48b3-89af-85ce2713e530-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.392051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.392018 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlgr\" (UniqueName: \"kubernetes.io/projected/53cb4a4a-12c5-48b3-89af-85ce2713e530-kube-api-access-9jlgr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.392214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.392157 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6nj8\" (UniqueName: \"kubernetes.io/projected/a88479da-823a-482b-8fb1-cb680c329d05-kube-api-access-n6nj8\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.392214 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.392159 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnbk\" (UniqueName: \"kubernetes.io/projected/89a00de5-a2dd-4a89-a10d-57993d24f88f-kube-api-access-sfnbk\") pod \"maas-api-key-cleanup-29892765-jr47n\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.479634 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.479602 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:00.487001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.486975 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:00.493074 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.493043 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:00.745245 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.745152 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:00.871547 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:45:00.871435 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cb4a4a_12c5_48b3_89af_85ce2713e530.slice/crio-e5cc30e3173122082ab1bc9dafe0d5cd7fa78fb13eac73a89fed067e2f92675b WatchSource:0}: Error finding container e5cc30e3173122082ab1bc9dafe0d5cd7fa78fb13eac73a89fed067e2f92675b: Status 404 returned error can't find the container with id e5cc30e3173122082ab1bc9dafe0d5cd7fa78fb13eac73a89fed067e2f92675b Nov 01 20:45:00.873718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.873685 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:00.877758 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.877732 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q"] Nov 01 20:45:00.919483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.919434 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2"] Nov 01 20:45:00.925292 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:45:00.925254 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a00de5_a2dd_4a89_a10d_57993d24f88f.slice/crio-c17ff16e1b4d32fdee07399f564a9d7be78a515fedc25b7cdad91616d68e4e83 WatchSource:0}: Error finding container c17ff16e1b4d32fdee07399f564a9d7be78a515fedc25b7cdad91616d68e4e83: Status 404 returned error can't find the container with id c17ff16e1b4d32fdee07399f564a9d7be78a515fedc25b7cdad91616d68e4e83 Nov 01 20:45:00.926012 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:00.925985 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n"] Nov 01 20:45:01.047277 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:01.047191 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-401f7dd59863: secret "agc-cred-401f7dd59863" not found Nov 01 20:45:01.047277 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:01.047223 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-63b1876e63ce: secret "agc-cred-63b1876e63ce" not found Nov 01 20:45:01.047277 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:01.047233 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9: [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:01.047563 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:01.047302 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials podName:47e4a7e2-cfd5-4459-8714-edf33451e2ac nodeName:}" failed. No retries permitted until 2026-11-01 20:45:03.047283092 +0000 UTC m=+2510.721542328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials") pod "payload-processing-external-model-6dcf58fd8-8dvn9" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac") : [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:01.147755 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:01.147707 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" event={"ID":"53cb4a4a-12c5-48b3-89af-85ce2713e530","Type":"ContainerStarted","Data":"e5cc30e3173122082ab1bc9dafe0d5cd7fa78fb13eac73a89fed067e2f92675b"} Nov 01 20:45:01.149161 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:01.149133 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" event={"ID":"89a00de5-a2dd-4a89-a10d-57993d24f88f","Type":"ContainerStarted","Data":"c17ff16e1b4d32fdee07399f564a9d7be78a515fedc25b7cdad91616d68e4e83"} Nov 01 20:45:01.150453 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:01.150421 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" event={"ID":"a88479da-823a-482b-8fb1-cb680c329d05","Type":"ContainerStarted","Data":"1e6ed1b81b02b6ce6c7b5b96e85489989aa8d2af611f2fafc073b4b2079a45cc"} Nov 01 20:45:01.801073 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:01.801034 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:02.089973 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.089939 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:45:02.163938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.163893 5633 generic.go:356] "Generic (PLEG): container finished" podID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerID="f6021b57add471c358c87fe763b40e603557f1f5d23f70338fa79f54e20cc719" exitCode=60 Nov 01 20:45:02.164146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.163976 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" event={"ID":"53cb4a4a-12c5-48b3-89af-85ce2713e530","Type":"ContainerDied","Data":"f6021b57add471c358c87fe763b40e603557f1f5d23f70338fa79f54e20cc719"} Nov 01 20:45:02.164720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.164692 5633 scope.go:122] "RemoveContainer" containerID="f6021b57add471c358c87fe763b40e603557f1f5d23f70338fa79f54e20cc719" Nov 01 20:45:02.172699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.172661 5633 generic.go:356] "Generic (PLEG): container finished" podID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerID="1e9ee1d12b8a332c5e98998a6cf06008a81eaf43b9f992f5138f0a7dd1a75b57" exitCode=60 Nov 01 20:45:02.172928 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.172832 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" event={"ID":"89a00de5-a2dd-4a89-a10d-57993d24f88f","Type":"ContainerDied","Data":"1e9ee1d12b8a332c5e98998a6cf06008a81eaf43b9f992f5138f0a7dd1a75b57"} Nov 01 20:45:02.173512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.173491 5633 scope.go:122] "RemoveContainer" containerID="1e9ee1d12b8a332c5e98998a6cf06008a81eaf43b9f992f5138f0a7dd1a75b57" Nov 01 20:45:02.196542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.196235 5633 generic.go:356] "Generic (PLEG): container finished" podID="a88479da-823a-482b-8fb1-cb680c329d05" containerID="8e3657dc3afda8956f4f83f2e7b6b8d8a391c91c15fc4db44268df29b1f77759" exitCode=60 Nov 01 20:45:02.196542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.196356 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" event={"ID":"a88479da-823a-482b-8fb1-cb680c329d05","Type":"ContainerDied","Data":"8e3657dc3afda8956f4f83f2e7b6b8d8a391c91c15fc4db44268df29b1f77759"} Nov 01 20:45:02.196798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:02.196703 5633 scope.go:122] "RemoveContainer" containerID="8e3657dc3afda8956f4f83f2e7b6b8d8a391c91c15fc4db44268df29b1f77759" Nov 01 20:45:03.071697 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:03.071660 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-401f7dd59863: secret "agc-cred-401f7dd59863" not found Nov 01 20:45:03.071697 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:03.071697 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-63b1876e63ce: secret "agc-cred-63b1876e63ce" not found Nov 01 20:45:03.071697 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:03.071707 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9: [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:03.072215 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:03.071768 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials podName:47e4a7e2-cfd5-4459-8714-edf33451e2ac nodeName:}" failed. No retries permitted until 2026-11-01 20:45:07.071749584 +0000 UTC m=+2514.746008821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials") pod "payload-processing-external-model-6dcf58fd8-8dvn9" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac") : [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:03.205893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.205855 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" event={"ID":"89a00de5-a2dd-4a89-a10d-57993d24f88f","Type":"ContainerStarted","Data":"7ff20f4016db290de68b333994f7ecbf2013332c427e71aa1dde90a99878a125"} Nov 01 20:45:03.207821 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.207787 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" event={"ID":"a88479da-823a-482b-8fb1-cb680c329d05","Type":"ContainerStarted","Data":"3838949897a6f83ff197b2cf7ee0352c824cb0f56b21bcb3aec5bfe881f808e6"} Nov 01 20:45:03.210072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.210043 5633 generic.go:356] "Generic (PLEG): container finished" podID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerID="c81bebffba583264d67b45b6a8414833404bc663cb31e70754db13d25a90503b" exitCode=60 Nov 01 20:45:03.210207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.210094 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" event={"ID":"53cb4a4a-12c5-48b3-89af-85ce2713e530","Type":"ContainerDied","Data":"c81bebffba583264d67b45b6a8414833404bc663cb31e70754db13d25a90503b"} Nov 01 20:45:03.210207 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.210141 5633 scope.go:122] "RemoveContainer" containerID="f6021b57add471c358c87fe763b40e603557f1f5d23f70338fa79f54e20cc719" Nov 01 20:45:03.210381 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.210365 5633 scope.go:122] "RemoveContainer" containerID="c81bebffba583264d67b45b6a8414833404bc663cb31e70754db13d25a90503b" Nov 01 20:45:03.210665 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:03.210639 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q_odh-ai-gateway-infra(53cb4a4a-12c5-48b3-89af-85ce2713e530)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" Nov 01 20:45:03.319577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.319521 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" podStartSLOduration=3.319505238 podStartE2EDuration="3.319505238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:03.317185874 +0000 UTC m=+2510.991445128" watchObservedRunningTime="2026-11-01 20:45:03.319505238 +0000 UTC m=+2510.993764492" Nov 01 20:45:03.432461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:03.432392 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" podStartSLOduration=3.432373601 podStartE2EDuration="3.432373601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:03.428919985 +0000 UTC m=+2511.103179240" watchObservedRunningTime="2026-11-01 20:45:03.432373601 +0000 UTC m=+2511.106632856" Nov 01 20:45:04.034462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.028230 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:04.071085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.071054 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9"] Nov 01 20:45:04.152299 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.152267 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:04.224779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.224742 5633 scope.go:122] "RemoveContainer" containerID="c81bebffba583264d67b45b6a8414833404bc663cb31e70754db13d25a90503b" Nov 01 20:45:04.225217 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:04.225188 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q_odh-ai-gateway-infra(53cb4a4a-12c5-48b3-89af-85ce2713e530)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" Nov 01 20:45:04.226790 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.226759 5633 generic.go:356] "Generic (PLEG): container finished" podID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerID="7ff20f4016db290de68b333994f7ecbf2013332c427e71aa1dde90a99878a125" exitCode=60 Nov 01 20:45:04.226897 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.226830 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" event={"ID":"89a00de5-a2dd-4a89-a10d-57993d24f88f","Type":"ContainerDied","Data":"7ff20f4016db290de68b333994f7ecbf2013332c427e71aa1dde90a99878a125"} Nov 01 20:45:04.227007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.226925 5633 scope.go:122] "RemoveContainer" containerID="1e9ee1d12b8a332c5e98998a6cf06008a81eaf43b9f992f5138f0a7dd1a75b57" Nov 01 20:45:04.227096 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.227073 5633 scope.go:122] "RemoveContainer" containerID="7ff20f4016db290de68b333994f7ecbf2013332c427e71aa1dde90a99878a125" Nov 01 20:45:04.227427 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:04.227402 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-jr47n_odh-ai-gateway-infra(89a00de5-a2dd-4a89-a10d-57993d24f88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" Nov 01 20:45:04.228992 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.228966 5633 generic.go:356] "Generic (PLEG): container finished" podID="a88479da-823a-482b-8fb1-cb680c329d05" containerID="3838949897a6f83ff197b2cf7ee0352c824cb0f56b21bcb3aec5bfe881f808e6" exitCode=60 Nov 01 20:45:04.229085 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.229019 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" event={"ID":"a88479da-823a-482b-8fb1-cb680c329d05","Type":"ContainerDied","Data":"3838949897a6f83ff197b2cf7ee0352c824cb0f56b21bcb3aec5bfe881f808e6"} Nov 01 20:45:04.229260 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.229247 5633 scope.go:122] "RemoveContainer" containerID="3838949897a6f83ff197b2cf7ee0352c824cb0f56b21bcb3aec5bfe881f808e6" Nov 01 20:45:04.229535 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:04.229519 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2_odh-ai-gateway-infra(a88479da-823a-482b-8fb1-cb680c329d05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" podUID="a88479da-823a-482b-8fb1-cb680c329d05" Nov 01 20:45:04.239832 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.239800 5633 scope.go:122] "RemoveContainer" containerID="8e3657dc3afda8956f4f83f2e7b6b8d8a391c91c15fc4db44268df29b1f77759" Nov 01 20:45:04.322841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:04.322752 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:05.237720 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:05.237689 5633 scope.go:122] "RemoveContainer" containerID="7ff20f4016db290de68b333994f7ecbf2013332c427e71aa1dde90a99878a125" Nov 01 20:45:05.238171 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:05.237985 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-jr47n_odh-ai-gateway-infra(89a00de5-a2dd-4a89-a10d-57993d24f88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" Nov 01 20:45:05.239306 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:05.239289 5633 scope.go:122] "RemoveContainer" containerID="3838949897a6f83ff197b2cf7ee0352c824cb0f56b21bcb3aec5bfe881f808e6" Nov 01 20:45:05.239577 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:05.239559 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2_odh-ai-gateway-infra(a88479da-823a-482b-8fb1-cb680c329d05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" podUID="a88479da-823a-482b-8fb1-cb680c329d05" Nov 01 20:45:05.471718 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:05.471669 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:06.472346 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:06.472298 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:07.124850 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:07.124814 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-401f7dd59863: secret "agc-cred-401f7dd59863" not found Nov 01 20:45:07.124850 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:07.124844 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4ca90c301433: secret "agc-cred-4ca90c301433" not found Nov 01 20:45:07.124850 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:07.124853 5633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-63b1876e63ce: secret "agc-cred-63b1876e63ce" not found Nov 01 20:45:07.125132 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:07.124866 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9: [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-4ca90c301433" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:07.125132 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:07.124921 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials podName:47e4a7e2-cfd5-4459-8714-edf33451e2ac nodeName:}" failed. No retries permitted until 2026-11-01 20:45:15.124905262 +0000 UTC m=+2522.799164509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials") pod "payload-processing-external-model-6dcf58fd8-8dvn9" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac") : [secret "agc-cred-401f7dd59863" not found, secret "agc-cred-4ca90c301433" not found, secret "agc-cred-63b1876e63ce" not found] Nov 01 20:45:09.072010 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.071915 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" podUID="47e4a7e2-cfd5-4459-8714-edf33451e2ac" containerName="payload-processing" containerID="cri-o://641fd16bed56eb6c5854ac024dd3af2aa030868b57618598f55a6e24c839b66b" gracePeriod=25 Nov 01 20:45:09.279119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.279073 5633 generic.go:356] "Generic (PLEG): container finished" podID="47e4a7e2-cfd5-4459-8714-edf33451e2ac" containerID="641fd16bed56eb6c5854ac024dd3af2aa030868b57618598f55a6e24c839b66b" exitCode=0 Nov 01 20:45:09.279408 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.279381 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" event={"ID":"47e4a7e2-cfd5-4459-8714-edf33451e2ac","Type":"ContainerDied","Data":"641fd16bed56eb6c5854ac024dd3af2aa030868b57618598f55a6e24c839b66b"} Nov 01 20:45:09.415270 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.415233 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:45:09.559959 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.559919 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-routing-overlay-volume\") pod \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " Nov 01 20:45:09.560257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.560235 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-kube-api-access-2zndw\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-kube-api-access-2zndw\") pod \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " Nov 01 20:45:09.560534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.560513 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials\") pod \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " Nov 01 20:45:09.560737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.560717 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-plugins-config-volume\") pod \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\" (UID: \"47e4a7e2-cfd5-4459-8714-edf33451e2ac\") " Nov 01 20:45:09.562947 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.562918 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-routing-overlay-volume" pod "47e4a7e2-cfd5-4459-8714-edf33451e2ac" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:09.565187 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.565149 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-plugins-config-volume" pod "47e4a7e2-cfd5-4459-8714-edf33451e2ac" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:09.568098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.568058 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-kube-api-access-2zndw" pod "47e4a7e2-cfd5-4459-8714-edf33451e2ac" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac"). InnerVolumeSpecName "kube-api-access-2zndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:09.568678 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.568644 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials" pod "47e4a7e2-cfd5-4459-8714-edf33451e2ac" (UID: "47e4a7e2-cfd5-4459-8714-edf33451e2ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:09.663866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.663746 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:09.663866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.663792 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zndw\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-kube-api-access-2zndw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:09.663866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.663810 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47e4a7e2-cfd5-4459-8714-edf33451e2ac-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:09.663866 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:09.663827 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e4a7e2-cfd5-4459-8714-edf33451e2ac-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:10.312544 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:10.312500 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" event={"ID":"47e4a7e2-cfd5-4459-8714-edf33451e2ac","Type":"ContainerDied","Data":"09fff9e0e2e1cf72f2dff9683d1e09fe55b6bb12583a5b6dd35dd0815bd01ae2"} Nov 01 20:45:10.313237 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:10.313217 5633 scope.go:122] "RemoveContainer" containerID="641fd16bed56eb6c5854ac024dd3af2aa030868b57618598f55a6e24c839b66b" Nov 01 20:45:10.321445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:10.313150 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9" Nov 01 20:45:10.521357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:10.519881 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9"] Nov 01 20:45:10.576527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:10.576320 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf58fd8-8dvn9"] Nov 01 20:45:10.733387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:10.731459 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4"] Nov 01 20:45:11.147355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:11.147286 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:11.272353 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:11.271893 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:11.533697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:11.533657 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:11.745445 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:11.740321 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:12.601190 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:12.601152 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e4a7e2-cfd5-4459-8714-edf33451e2ac" path="/var/lib/kubelet/pods/47e4a7e2-cfd5-4459-8714-edf33451e2ac/volumes" Nov 01 20:45:12.742363 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:12.740237 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:45:14.460218 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:14.460180 5633 scope.go:122] "RemoveContainer" containerID="c81bebffba583264d67b45b6a8414833404bc663cb31e70754db13d25a90503b" Nov 01 20:45:14.460752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:14.460404 5633 scope.go:122] "RemoveContainer" containerID="7ff20f4016db290de68b333994f7ecbf2013332c427e71aa1dde90a99878a125" Nov 01 20:45:15.459996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.459963 5633 scope.go:122] "RemoveContainer" containerID="3838949897a6f83ff197b2cf7ee0352c824cb0f56b21bcb3aec5bfe881f808e6" Nov 01 20:45:15.571070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.570970 5633 generic.go:356] "Generic (PLEG): container finished" podID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerID="d855438599918c10ee68810d05a1554559845815993574fa8a85973cc073d5b7" exitCode=60 Nov 01 20:45:15.571070 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.571048 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" event={"ID":"53cb4a4a-12c5-48b3-89af-85ce2713e530","Type":"ContainerDied","Data":"d855438599918c10ee68810d05a1554559845815993574fa8a85973cc073d5b7"} Nov 01 20:45:15.571612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.571113 5633 scope.go:122] "RemoveContainer" containerID="c81bebffba583264d67b45b6a8414833404bc663cb31e70754db13d25a90503b" Nov 01 20:45:15.571612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.571536 5633 scope.go:122] "RemoveContainer" containerID="d855438599918c10ee68810d05a1554559845815993574fa8a85973cc073d5b7" Nov 01 20:45:15.571884 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:15.571857 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q_odh-ai-gateway-infra(53cb4a4a-12c5-48b3-89af-85ce2713e530)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" Nov 01 20:45:15.573477 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.573455 5633 generic.go:356] "Generic (PLEG): container finished" podID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerID="a0276fdff82bf5d47f64f95b79732cae4fc58e58b4275c7e10a76c1362c745fa" exitCode=60 Nov 01 20:45:15.573597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.573534 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" event={"ID":"89a00de5-a2dd-4a89-a10d-57993d24f88f","Type":"ContainerDied","Data":"a0276fdff82bf5d47f64f95b79732cae4fc58e58b4275c7e10a76c1362c745fa"} Nov 01 20:45:15.573915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.573893 5633 scope.go:122] "RemoveContainer" containerID="a0276fdff82bf5d47f64f95b79732cae4fc58e58b4275c7e10a76c1362c745fa" Nov 01 20:45:15.574172 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:15.574157 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-jr47n_odh-ai-gateway-infra(89a00de5-a2dd-4a89-a10d-57993d24f88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" Nov 01 20:45:15.585533 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.585511 5633 scope.go:122] "RemoveContainer" containerID="7ff20f4016db290de68b333994f7ecbf2013332c427e71aa1dde90a99878a125" Nov 01 20:45:15.733048 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:15.733003 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" podUID="e346846b-8520-402b-87c2-84eb4d32e60f" containerName="payload-pre-processing" containerID="cri-o://ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661" gracePeriod=25 Nov 01 20:45:16.158489 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:16.158445 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:45:16.240605 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.240569 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:16.589660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.589627 5633 generic.go:356] "Generic (PLEG): container finished" podID="a88479da-823a-482b-8fb1-cb680c329d05" containerID="a466866a7bf8df1fe004d4b81293507b98ec26e71e338f9487cfc58ee93fe3d8" exitCode=60 Nov 01 20:45:16.590093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.589709 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" event={"ID":"a88479da-823a-482b-8fb1-cb680c329d05","Type":"ContainerDied","Data":"a466866a7bf8df1fe004d4b81293507b98ec26e71e338f9487cfc58ee93fe3d8"} Nov 01 20:45:16.590093 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.589786 5633 scope.go:122] "RemoveContainer" containerID="3838949897a6f83ff197b2cf7ee0352c824cb0f56b21bcb3aec5bfe881f808e6" Nov 01 20:45:16.590253 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.590235 5633 scope.go:122] "RemoveContainer" containerID="a466866a7bf8df1fe004d4b81293507b98ec26e71e338f9487cfc58ee93fe3d8" Nov 01 20:45:16.590632 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:16.590611 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2_odh-ai-gateway-infra(a88479da-823a-482b-8fb1-cb680c329d05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" podUID="a88479da-823a-482b-8fb1-cb680c329d05" Nov 01 20:45:16.699599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.699563 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q"] Nov 01 20:45:16.808226 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.808189 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n"] Nov 01 20:45:16.857420 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.856807 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:16.977350 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.977312 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:16.980360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.980312 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cb4a4a-12c5-48b3-89af-85ce2713e530-kube-api-access-9jlgr\" (UniqueName: \"kubernetes.io/projected/53cb4a4a-12c5-48b3-89af-85ce2713e530-kube-api-access-9jlgr\") pod \"53cb4a4a-12c5-48b3-89af-85ce2713e530\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " Nov 01 20:45:16.980545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.980513 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cb4a4a-12c5-48b3-89af-85ce2713e530-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4a4a-12c5-48b3-89af-85ce2713e530-maas-api-service-ca\") pod \"53cb4a4a-12c5-48b3-89af-85ce2713e530\" (UID: \"53cb4a4a-12c5-48b3-89af-85ce2713e530\") " Nov 01 20:45:16.980893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.980866 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cb4a4a-12c5-48b3-89af-85ce2713e530-maas-api-service-ca" pod "53cb4a4a-12c5-48b3-89af-85ce2713e530" (UID: "53cb4a4a-12c5-48b3-89af-85ce2713e530"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:16.981168 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.981005 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53cb4a4a-12c5-48b3-89af-85ce2713e530-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:16.982727 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:16.982696 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cb4a4a-12c5-48b3-89af-85ce2713e530-kube-api-access-9jlgr" pod "53cb4a4a-12c5-48b3-89af-85ce2713e530" (UID: "53cb4a4a-12c5-48b3-89af-85ce2713e530"). InnerVolumeSpecName "kube-api-access-9jlgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:17.047470 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.047436 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:17.082820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.081826 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a00de5-a2dd-4a89-a10d-57993d24f88f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a00de5-a2dd-4a89-a10d-57993d24f88f-maas-api-service-ca\") pod \"89a00de5-a2dd-4a89-a10d-57993d24f88f\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " Nov 01 20:45:17.082820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.081909 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a00de5-a2dd-4a89-a10d-57993d24f88f-kube-api-access-sfnbk\" (UniqueName: \"kubernetes.io/projected/89a00de5-a2dd-4a89-a10d-57993d24f88f-kube-api-access-sfnbk\") pod \"89a00de5-a2dd-4a89-a10d-57993d24f88f\" (UID: \"89a00de5-a2dd-4a89-a10d-57993d24f88f\") " Nov 01 20:45:17.082820 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.082432 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jlgr\" (UniqueName: \"kubernetes.io/projected/53cb4a4a-12c5-48b3-89af-85ce2713e530-kube-api-access-9jlgr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:17.091377 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.091269 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a00de5-a2dd-4a89-a10d-57993d24f88f-maas-api-service-ca" pod "89a00de5-a2dd-4a89-a10d-57993d24f88f" (UID: "89a00de5-a2dd-4a89-a10d-57993d24f88f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:17.094258 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.091905 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a00de5-a2dd-4a89-a10d-57993d24f88f-kube-api-access-sfnbk" pod "89a00de5-a2dd-4a89-a10d-57993d24f88f" (UID: "89a00de5-a2dd-4a89-a10d-57993d24f88f"). InnerVolumeSpecName "kube-api-access-sfnbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:17.183777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.183689 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89a00de5-a2dd-4a89-a10d-57993d24f88f-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:17.183777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.183721 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfnbk\" (UniqueName: \"kubernetes.io/projected/89a00de5-a2dd-4a89-a10d-57993d24f88f-kube-api-access-sfnbk\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:17.486796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.486754 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" podUID="e346846b-8520-402b-87c2-84eb4d32e60f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:45:17.598082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.598053 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" Nov 01 20:45:17.598613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.598084 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n" event={"ID":"89a00de5-a2dd-4a89-a10d-57993d24f88f","Type":"ContainerDied","Data":"c17ff16e1b4d32fdee07399f564a9d7be78a515fedc25b7cdad91616d68e4e83"} Nov 01 20:45:17.598613 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.598147 5633 scope.go:122] "RemoveContainer" containerID="a0276fdff82bf5d47f64f95b79732cae4fc58e58b4275c7e10a76c1362c745fa" Nov 01 20:45:17.601695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.601673 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" Nov 01 20:45:17.601695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.601682 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q" event={"ID":"53cb4a4a-12c5-48b3-89af-85ce2713e530","Type":"ContainerDied","Data":"e5cc30e3173122082ab1bc9dafe0d5cd7fa78fb13eac73a89fed067e2f92675b"} Nov 01 20:45:17.610587 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.610427 5633 scope.go:122] "RemoveContainer" containerID="d855438599918c10ee68810d05a1554559845815993574fa8a85973cc073d5b7" Nov 01 20:45:17.716651 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.716624 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2"] Nov 01 20:45:17.762019 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.761935 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n"] Nov 01 20:45:17.765895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.765865 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-jr47n"] Nov 01 20:45:17.869926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.869895 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:17.873999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.873967 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q"] Nov 01 20:45:17.877574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.877552 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc24d-29892765-hwn7q"] Nov 01 20:45:17.991517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.991480 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:17.995325 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.995297 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88479da-823a-482b-8fb1-cb680c329d05-kube-api-access-n6nj8\" (UniqueName: \"kubernetes.io/projected/a88479da-823a-482b-8fb1-cb680c329d05-kube-api-access-n6nj8\") pod \"a88479da-823a-482b-8fb1-cb680c329d05\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " Nov 01 20:45:17.995507 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.995458 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88479da-823a-482b-8fb1-cb680c329d05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88479da-823a-482b-8fb1-cb680c329d05-maas-api-service-ca\") pod \"a88479da-823a-482b-8fb1-cb680c329d05\" (UID: \"a88479da-823a-482b-8fb1-cb680c329d05\") " Nov 01 20:45:17.996084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.996056 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88479da-823a-482b-8fb1-cb680c329d05-maas-api-service-ca" pod "a88479da-823a-482b-8fb1-cb680c329d05" (UID: "a88479da-823a-482b-8fb1-cb680c329d05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:17.998502 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:17.998471 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88479da-823a-482b-8fb1-cb680c329d05-kube-api-access-n6nj8" pod "a88479da-823a-482b-8fb1-cb680c329d05" (UID: "a88479da-823a-482b-8fb1-cb680c329d05"). InnerVolumeSpecName "kube-api-access-n6nj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:18.097435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.097349 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6nj8\" (UniqueName: \"kubernetes.io/projected/a88479da-823a-482b-8fb1-cb680c329d05-kube-api-access-n6nj8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:18.097435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.097390 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88479da-823a-482b-8fb1-cb680c329d05-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:18.467876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.467840 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" path="/var/lib/kubelet/pods/53cb4a4a-12c5-48b3-89af-85ce2713e530/volumes" Nov 01 20:45:18.468259 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.468246 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" path="/var/lib/kubelet/pods/89a00de5-a2dd-4a89-a10d-57993d24f88f/volumes" Nov 01 20:45:18.610241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.610216 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" Nov 01 20:45:18.610690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.610249 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2" event={"ID":"a88479da-823a-482b-8fb1-cb680c329d05","Type":"ContainerDied","Data":"1e6ed1b81b02b6ce6c7b5b96e85489989aa8d2af611f2fafc073b4b2079a45cc"} Nov 01 20:45:18.610690 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.610309 5633 scope.go:122] "RemoveContainer" containerID="a466866a7bf8df1fe004d4b81293507b98ec26e71e338f9487cfc58ee93fe3d8" Nov 01 20:45:18.773673 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.773592 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2"] Nov 01 20:45:18.822693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:18.822645 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892765-j5jv2"] Nov 01 20:45:20.467313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:20.467277 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88479da-823a-482b-8fb1-cb680c329d05" path="/var/lib/kubelet/pods/a88479da-823a-482b-8fb1-cb680c329d05/volumes" Nov 01 20:45:22.558938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:22.558843 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:22.735172 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:22.735138 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:24.078997 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:24.078956 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:24.692152 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:24.692109 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4"] Nov 01 20:45:24.692737 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:24.692700 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" containerID="cri-o://c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c" gracePeriod=30 Nov 01 20:45:24.701205 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:24.701158 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:25.480630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:25.480566 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:27.486763 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:27.486723 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" podUID="e346846b-8520-402b-87c2-84eb4d32e60f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:45:27.792625 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:27.792527 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:28.902364 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:28.902314 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:45:29.892867 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.892839 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:45:29.936240 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936205 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-envoy\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.936712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936260 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-token\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-token\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.936712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936289 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00f1c85-71ca-4ca3-b151-c2407fb42882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f1c85-71ca-4ca3-b151-c2407fb42882-istiod-ca-cert\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.936712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936442 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-socket\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.936712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936523 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-data\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.936712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936550 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-kube-api-access-9hfp5\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-kube-api-access-9hfp5\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.936712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936607 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-certs\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.936712 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936652 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-credential-socket\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.937157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936722 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-podinfo\") pod \"f00f1c85-71ca-4ca3-b151-c2407fb42882\" (UID: \"f00f1c85-71ca-4ca3-b151-c2407fb42882\") " Nov 01 20:45:29.937157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936740 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-socket" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:29.937157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936753 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00f1c85-71ca-4ca3-b151-c2407fb42882-istiod-ca-cert" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:29.937157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936966 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-credential-socket" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:29.937157 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.936987 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-certs" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:29.937632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.937349 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:29.937632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.937373 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:29.937632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.937389 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:29.937632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.937404 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f1c85-71ca-4ca3-b151-c2407fb42882-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:29.937632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.937464 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-data" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:29.938759 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.938731 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-token" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:29.938893 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.938878 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-kube-api-access-9hfp5" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "kube-api-access-9hfp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:29.938941 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.938877 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-envoy" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:29.939177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:29.939164 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-podinfo" pod "f00f1c85-71ca-4ca3-b151-c2407fb42882" (UID: "f00f1c85-71ca-4ca3-b151-c2407fb42882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:45:30.038644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.038553 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:30.038644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.038589 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hfp5\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-kube-api-access-9hfp5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:30.038644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.038604 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:30.038644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.038616 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:30.038644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.038628 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f1c85-71ca-4ca3-b151-c2407fb42882-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:30.698189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.698157 5633 generic.go:356] "Generic (PLEG): container finished" podID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerID="c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c" exitCode=0 Nov 01 20:45:30.698419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.698208 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" Nov 01 20:45:30.698419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.698221 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" event={"ID":"f00f1c85-71ca-4ca3-b151-c2407fb42882","Type":"ContainerDied","Data":"c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c"} Nov 01 20:45:30.698419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.698264 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4" event={"ID":"f00f1c85-71ca-4ca3-b151-c2407fb42882","Type":"ContainerDied","Data":"a6486b3503ef1d7f870dcde276b0c7b2993326a28f1f2bdb1df251cedbe599bc"} Nov 01 20:45:30.698419 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.698285 5633 scope.go:122] "RemoveContainer" containerID="c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c" Nov 01 20:45:30.709126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.709102 5633 scope.go:122] "RemoveContainer" containerID="61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8" Nov 01 20:45:30.719612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.719590 5633 scope.go:122] "RemoveContainer" containerID="c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c" Nov 01 20:45:30.719933 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:30.719912 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c\": container with ID starting with c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c not found: ID does not exist" containerID="c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c" Nov 01 20:45:30.720020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.719945 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c"} err="failed to get container status \"c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c\": rpc error: code = NotFound desc = could not find container \"c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c\": container with ID starting with c46c28edef48c3b1b6eb9f9d5f82a8f5b4caf2fb4dc1c19f2b807f949adb9f8c not found: ID does not exist" Nov 01 20:45:30.720020 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.719966 5633 scope.go:122] "RemoveContainer" containerID="61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8" Nov 01 20:45:30.720209 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:30.720191 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8\": container with ID starting with 61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8 not found: ID does not exist" containerID="61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8" Nov 01 20:45:30.720255 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.720216 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8"} err="failed to get container status \"61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8\": rpc error: code = NotFound desc = could not find container \"61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8\": container with ID starting with 61067ab356945c7d8877917828510c31ce21469a36cd790b73ae0da1067297d8 not found: ID does not exist" Nov 01 20:45:30.911189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.911101 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4"] Nov 01 20:45:30.961084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:30.961049 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-pjlb4"] Nov 01 20:45:31.305490 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:31.305454 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:31.894999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:31.894958 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:32.470810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:32.470777 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" path="/var/lib/kubelet/pods/f00f1c85-71ca-4ca3-b151-c2407fb42882/volumes" Nov 01 20:45:32.567869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:32.567822 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:34.629768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:34.629720 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn"] Nov 01 20:45:34.744123 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:34.744077 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl"] Nov 01 20:45:34.979057 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:34.979019 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:35.102451 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:35.102419 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:36.002836 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.002764 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:45:36.133966 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.133921 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e346846b-8520-402b-87c2-84eb4d32e60f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e346846b-8520-402b-87c2-84eb4d32e60f-plugins-config-volume\") pod \"e346846b-8520-402b-87c2-84eb4d32e60f\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " Nov 01 20:45:36.134177 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.133983 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e346846b-8520-402b-87c2-84eb4d32e60f-kube-api-access-r29xd\" (UniqueName: \"kubernetes.io/projected/e346846b-8520-402b-87c2-84eb4d32e60f-kube-api-access-r29xd\") pod \"e346846b-8520-402b-87c2-84eb4d32e60f\" (UID: \"e346846b-8520-402b-87c2-84eb4d32e60f\") " Nov 01 20:45:36.134435 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.134406 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e346846b-8520-402b-87c2-84eb4d32e60f-plugins-config-volume" pod "e346846b-8520-402b-87c2-84eb4d32e60f" (UID: "e346846b-8520-402b-87c2-84eb4d32e60f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:36.136320 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.136290 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e346846b-8520-402b-87c2-84eb4d32e60f-kube-api-access-r29xd" pod "e346846b-8520-402b-87c2-84eb4d32e60f" (UID: "e346846b-8520-402b-87c2-84eb4d32e60f"). InnerVolumeSpecName "kube-api-access-r29xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:36.236198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.236151 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r29xd\" (UniqueName: \"kubernetes.io/projected/e346846b-8520-402b-87c2-84eb4d32e60f-kube-api-access-r29xd\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:36.236198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.236195 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e346846b-8520-402b-87c2-84eb4d32e60f-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:36.738458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.738372 5633 generic.go:356] "Generic (PLEG): container finished" podID="e346846b-8520-402b-87c2-84eb4d32e60f" containerID="ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661" exitCode=0 Nov 01 20:45:36.738458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.738425 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" Nov 01 20:45:36.738646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.738464 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" event={"ID":"e346846b-8520-402b-87c2-84eb4d32e60f","Type":"ContainerDied","Data":"ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661"} Nov 01 20:45:36.738646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.738523 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4" event={"ID":"e346846b-8520-402b-87c2-84eb4d32e60f","Type":"ContainerDied","Data":"8d789ac2fb915cbe3d597b3279c59c8702f4dfb134341b0ac9f99e7b04ffd13d"} Nov 01 20:45:36.738646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.738545 5633 scope.go:122] "RemoveContainer" containerID="ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661" Nov 01 20:45:36.748610 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.748588 5633 scope.go:122] "RemoveContainer" containerID="ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661" Nov 01 20:45:36.748913 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:36.748892 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661\": container with ID starting with ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661 not found: ID does not exist" containerID="ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661" Nov 01 20:45:36.748984 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.748926 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661"} err="failed to get container status \"ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661\": rpc error: code = NotFound desc = could not find container \"ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661\": container with ID starting with ce387f5bc5296fad6a88d3f554d7693b7406e478d908166a3635e5c9189c0661 not found: ID does not exist" Nov 01 20:45:36.903983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.903949 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4"] Nov 01 20:45:36.951750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:36.951713 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-df41b8-b5f9846f5-8pjd4"] Nov 01 20:45:37.487783 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:37.487746 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:38.467216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:38.467177 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e346846b-8520-402b-87c2-84eb4d32e60f" path="/var/lib/kubelet/pods/e346846b-8520-402b-87c2-84eb4d32e60f/volumes" Nov 01 20:45:39.630833 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:39.630773 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" podUID="0fe286da-6465-4d55-b062-0af4426ebf20" containerName="payload-processing" containerID="cri-o://10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1" gracePeriod=25 Nov 01 20:45:39.744547 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:39.744499 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" podUID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" containerName="payload-pre-processing" containerID="cri-o://6c999acebff5dd60b2fe244ecd08180c1e3f9974e939231ae089e9f10364c26b" gracePeriod=25 Nov 01 20:45:42.186644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:42.186605 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:43.340021 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.339981 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64"] Nov 01 20:45:43.340512 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340497 5633 cpu_manager.go:414] "containerMap: removing container" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:45:43.340574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340513 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:45:43.340574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340534 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" Nov 01 20:45:43.340574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340539 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" Nov 01 20:45:43.340574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340553 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:45:43.340574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340558 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:45:43.340574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340569 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:45:43.340574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340573 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340582 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e346846b-8520-402b-87c2-84eb4d32e60f" containerName="payload-pre-processing" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340587 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e346846b-8520-402b-87c2-84eb4d32e60f" containerName="payload-pre-processing" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340610 5633 cpu_manager.go:414] "containerMap: removing container" podUID="47e4a7e2-cfd5-4459-8714-edf33451e2ac" containerName="payload-processing" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340615 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e4a7e2-cfd5-4459-8714-edf33451e2ac" containerName="payload-processing" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340623 5633 cpu_manager.go:414] "containerMap: removing container" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340627 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340638 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340643 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340650 5633 cpu_manager.go:414] "containerMap: removing container" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340654 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340661 5633 cpu_manager.go:414] "containerMap: removing container" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340667 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340745 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340753 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340758 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e346846b-8520-402b-87c2-84eb4d32e60f" containerName="payload-pre-processing" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340764 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e4a7e2-cfd5-4459-8714-edf33451e2ac" containerName="payload-processing" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340770 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340776 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340780 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340786 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340791 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340798 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:45:43.340797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.340804 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00f1c85-71ca-4ca3-b151-c2407fb42882" containerName="istio-proxy" Nov 01 20:45:43.345225 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.345206 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.400745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.400716 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64"] Nov 01 20:45:43.445413 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.445384 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:45:43.446578 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.446553 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:43.446695 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.446584 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:45:43.446770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.446749 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:45:43.524354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.524300 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqczn\" (UniqueName: \"kubernetes.io/projected/f7f2597c-78ef-41a7-afd2-0686614f8925-kube-api-access-pqczn\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.524558 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.524528 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.524697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.524678 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.625802 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.625712 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.625986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.625825 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.625986 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.625865 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqczn\" (UniqueName: \"kubernetes.io/projected/f7f2597c-78ef-41a7-afd2-0686614f8925-kube-api-access-pqczn\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.626545 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.626521 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.628497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.628467 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.682653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.682617 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqczn\" (UniqueName: \"kubernetes.io/projected/f7f2597c-78ef-41a7-afd2-0686614f8925-kube-api-access-pqczn\") pod \"maas-api-revoke-sub-d31bbeaf-csg64\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:43.958733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:43.958647 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:44.339809 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:45:44.339779 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f2597c_78ef_41a7_afd2_0686614f8925.slice/crio-6838d91f5c3072014c8b3e40a4c9dca937ac9e0cec8a82cc7bcd4294a2932d41 WatchSource:0}: Error finding container 6838d91f5c3072014c8b3e40a4c9dca937ac9e0cec8a82cc7bcd4294a2932d41: Status 404 returned error can't find the container with id 6838d91f5c3072014c8b3e40a4c9dca937ac9e0cec8a82cc7bcd4294a2932d41 Nov 01 20:45:44.339923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:44.339846 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64"] Nov 01 20:45:44.801975 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:44.801928 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" event={"ID":"f7f2597c-78ef-41a7-afd2-0686614f8925","Type":"ContainerStarted","Data":"14222b65d710fd15fbf06069e1a078cb6db78f9819fe89040087263ceba5409b"} Nov 01 20:45:44.802436 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:44.802024 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" event={"ID":"f7f2597c-78ef-41a7-afd2-0686614f8925","Type":"ContainerStarted","Data":"6838d91f5c3072014c8b3e40a4c9dca937ac9e0cec8a82cc7bcd4294a2932d41"} Nov 01 20:45:44.916806 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:44.916704 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" podStartSLOduration=1.9166887419999998 podStartE2EDuration="1.916688742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:44.914584168 +0000 UTC m=+2552.588843436" watchObservedRunningTime="2026-11-01 20:45:44.916688742 +0000 UTC m=+2552.590947996" Nov 01 20:45:45.415446 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:45.415404 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:45.809767 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:45.809734 5633 generic.go:356] "Generic (PLEG): container finished" podID="f7f2597c-78ef-41a7-afd2-0686614f8925" containerID="14222b65d710fd15fbf06069e1a078cb6db78f9819fe89040087263ceba5409b" exitCode=0 Nov 01 20:45:45.810232 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:45.809831 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" event={"ID":"f7f2597c-78ef-41a7-afd2-0686614f8925","Type":"ContainerDied","Data":"14222b65d710fd15fbf06069e1a078cb6db78f9819fe89040087263ceba5409b"} Nov 01 20:45:46.156297 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:46.156211 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:45:46.380815 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:46.380764 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:46.970097 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:46.970066 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:47.011223 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.011185 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" podUID="0fe286da-6465-4d55-b062-0af4426ebf20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:45:47.011447 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.011189 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" podUID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:45:47.067312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.067226 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-xks-ca\") pod \"f7f2597c-78ef-41a7-afd2-0686614f8925\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " Nov 01 20:45:47.067312 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.067293 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-openshift-ca\") pod \"f7f2597c-78ef-41a7-afd2-0686614f8925\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " Nov 01 20:45:47.067597 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.067406 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f2597c-78ef-41a7-afd2-0686614f8925-kube-api-access-pqczn\" (UniqueName: \"kubernetes.io/projected/f7f2597c-78ef-41a7-afd2-0686614f8925-kube-api-access-pqczn\") pod \"f7f2597c-78ef-41a7-afd2-0686614f8925\" (UID: \"f7f2597c-78ef-41a7-afd2-0686614f8925\") " Nov 01 20:45:47.067663 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.067644 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-openshift-ca" pod "f7f2597c-78ef-41a7-afd2-0686614f8925" (UID: "f7f2597c-78ef-41a7-afd2-0686614f8925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:47.067879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.067852 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:47.069668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.069642 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f2597c-78ef-41a7-afd2-0686614f8925-kube-api-access-pqczn" pod "f7f2597c-78ef-41a7-afd2-0686614f8925" (UID: "f7f2597c-78ef-41a7-afd2-0686614f8925"). InnerVolumeSpecName "kube-api-access-pqczn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:47.069668 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.069658 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-xks-ca" pod "f7f2597c-78ef-41a7-afd2-0686614f8925" (UID: "f7f2597c-78ef-41a7-afd2-0686614f8925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:47.169262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.169219 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqczn\" (UniqueName: \"kubernetes.io/projected/f7f2597c-78ef-41a7-afd2-0686614f8925-kube-api-access-pqczn\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:47.169262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.169262 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f2597c-78ef-41a7-afd2-0686614f8925-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:47.823409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.823372 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" Nov 01 20:45:47.823409 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.823390 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64" event={"ID":"f7f2597c-78ef-41a7-afd2-0686614f8925","Type":"ContainerDied","Data":"6838d91f5c3072014c8b3e40a4c9dca937ac9e0cec8a82cc7bcd4294a2932d41"} Nov 01 20:45:47.823666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:47.823445 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6838d91f5c3072014c8b3e40a4c9dca937ac9e0cec8a82cc7bcd4294a2932d41" Nov 01 20:45:48.090553 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:48.090464 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:50.991645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:50.991604 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:45:51.113792 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:51.113760 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:51.248655 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:51.248568 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:52.590878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:52.590824 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:52.845491 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:52.845406 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:53.061948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:53.061916 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:53.071976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:53.071951 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh"] Nov 01 20:45:53.072173 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:53.072145 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" containerID="cri-o://41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f" gracePeriod=30 Nov 01 20:45:55.437721 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:55.437686 5633 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:45:55.438131 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:55.437744 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:45:55.827899 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:55.827860 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:55.947256 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:55.947220 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:56.350885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:56.350844 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:56.833241 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:56.833203 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:56.963163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:56.963120 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:57.011262 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:57.011222 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" podUID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:45:57.011460 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:57.011222 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" podUID="0fe286da-6465-4d55-b062-0af4426ebf20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:45:57.236181 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:57.236131 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:57.376680 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:57.376648 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:57.583084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:57.583000 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:57.894741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:57.894639 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:58.340885 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.340857 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:45:58.399824 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.399789 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-envoy\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.399878 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-data\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.399904 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-credential-socket\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.399928 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cd787f-33ca-49ad-968b-d86b755b5826-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cd787f-33ca-49ad-968b-d86b755b5826-istiod-ca-cert\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.399956 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-certs\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.399980 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-istio-token\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-istio-token\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400105 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-socket\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400163 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-kube-api-access-h947b\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-kube-api-access-h947b\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400215 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41cd787f-33ca-49ad-968b-d86b755b5826-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cd787f-33ca-49ad-968b-d86b755b5826-istio-podinfo\") pod \"41cd787f-33ca-49ad-968b-d86b755b5826\" (UID: \"41cd787f-33ca-49ad-968b-d86b755b5826\") " Nov 01 20:45:58.400303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400247 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-data" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:58.400303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400255 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-credential-socket" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:58.400606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400404 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cd787f-33ca-49ad-968b-d86b755b5826-istiod-ca-cert" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:58.400606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400542 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-socket" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:58.400777 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400321 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-certs" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:58.400919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400812 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.400919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400830 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.400919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400842 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.400919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.400851 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cd787f-33ca-49ad-968b-d86b755b5826-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.402670 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.402630 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-envoy" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:45:58.402993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.402903 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-kube-api-access-h947b" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "kube-api-access-h947b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:58.403235 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.403094 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41cd787f-33ca-49ad-968b-d86b755b5826-istio-podinfo" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:45:58.403647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.403624 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-istio-token" pod "41cd787f-33ca-49ad-968b-d86b755b5826" (UID: "41cd787f-33ca-49ad-968b-d86b755b5826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:58.455964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.455881 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:58.502246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.502213 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h947b\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-kube-api-access-h947b\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.502246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.502245 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cd787f-33ca-49ad-968b-d86b755b5826-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.502483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.502255 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.502483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.502264 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cd787f-33ca-49ad-968b-d86b755b5826-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.502483 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.502273 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cd787f-33ca-49ad-968b-d86b755b5826-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:58.621069 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.621024 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:45:58.906996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.906957 5633 generic.go:356] "Generic (PLEG): container finished" podID="41cd787f-33ca-49ad-968b-d86b755b5826" containerID="41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f" exitCode=0 Nov 01 20:45:58.907521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.907025 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" Nov 01 20:45:58.907521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.907042 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" event={"ID":"41cd787f-33ca-49ad-968b-d86b755b5826","Type":"ContainerDied","Data":"41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f"} Nov 01 20:45:58.907521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.907113 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh" event={"ID":"41cd787f-33ca-49ad-968b-d86b755b5826","Type":"ContainerDied","Data":"d0695788e851c68161d52c084cadb4f7b348e6769644a1c1bf33dfbc7bacbc19"} Nov 01 20:45:58.907521 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.907152 5633 scope.go:122] "RemoveContainer" containerID="41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f" Nov 01 20:45:58.917601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.917583 5633 scope.go:122] "RemoveContainer" containerID="5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274" Nov 01 20:45:58.927396 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.927373 5633 scope.go:122] "RemoveContainer" containerID="41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f" Nov 01 20:45:58.927733 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:58.927713 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f\": container with ID starting with 41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f not found: ID does not exist" containerID="41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f" Nov 01 20:45:58.927782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.927744 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f"} err="failed to get container status \"41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f\": rpc error: code = NotFound desc = could not find container \"41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f\": container with ID starting with 41f4428c40fbe5c5c2d534f2aebefe89bd256d6137ca54a5ef92853c2aa99b3f not found: ID does not exist" Nov 01 20:45:58.927782 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.927765 5633 scope.go:122] "RemoveContainer" containerID="5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274" Nov 01 20:45:58.928075 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:58.928057 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274\": container with ID starting with 5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274 not found: ID does not exist" containerID="5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274" Nov 01 20:45:58.928126 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:58.928081 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274"} err="failed to get container status \"5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274\": rpc error: code = NotFound desc = could not find container \"5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274\": container with ID starting with 5bf33b3b47f6f1175e35e1d09c4ed06b68e4539a1b610528a577ca5f4a09a274 not found: ID does not exist" Nov 01 20:45:59.071441 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.071402 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh"] Nov 01 20:45:59.074911 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.074872 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-gdchh"] Nov 01 20:45:59.393918 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.393878 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:45:59.911751 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.911667 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:45:59.914818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.914789 5633 generic.go:356] "Generic (PLEG): container finished" podID="0fe286da-6465-4d55-b062-0af4426ebf20" containerID="10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1" exitCode=0 Nov 01 20:45:59.914971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.914837 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" Nov 01 20:45:59.914971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.914864 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" event={"ID":"0fe286da-6465-4d55-b062-0af4426ebf20","Type":"ContainerDied","Data":"10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1"} Nov 01 20:45:59.914971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.914913 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn" event={"ID":"0fe286da-6465-4d55-b062-0af4426ebf20","Type":"ContainerDied","Data":"a686bfaa2a9744323a24972f5345416e0f504dfc35e470a7fb50a61ecdb431c2"} Nov 01 20:45:59.914971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.914939 5633 scope.go:122] "RemoveContainer" containerID="10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1" Nov 01 20:45:59.917938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.917913 5633 generic.go:356] "Generic (PLEG): container finished" podID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" containerID="6c999acebff5dd60b2fe244ecd08180c1e3f9974e939231ae089e9f10364c26b" exitCode=0 Nov 01 20:45:59.918072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.917985 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" event={"ID":"61a96f92-3084-4be6-b7a8-7f5f89c257e4","Type":"ContainerDied","Data":"6c999acebff5dd60b2fe244ecd08180c1e3f9974e939231ae089e9f10364c26b"} Nov 01 20:45:59.926732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.925757 5633 scope.go:122] "RemoveContainer" containerID="10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1" Nov 01 20:45:59.926732 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:45:59.926145 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1\": container with ID starting with 10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1 not found: ID does not exist" containerID="10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1" Nov 01 20:45:59.926732 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:45:59.926180 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1"} err="failed to get container status \"10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1\": rpc error: code = NotFound desc = could not find container \"10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1\": container with ID starting with 10826fa774e1b299b086492d8d4c17ba902acc72ae40762ccce6c884721903c1 not found: ID does not exist" Nov 01 20:46:00.013510 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.013485 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:46:00.021127 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.021105 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe286da-6465-4d55-b062-0af4426ebf20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe286da-6465-4d55-b062-0af4426ebf20-plugins-config-volume\") pod \"0fe286da-6465-4d55-b062-0af4426ebf20\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " Nov 01 20:46:00.021252 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.021140 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe286da-6465-4d55-b062-0af4426ebf20-kube-api-access-t9lj6\" (UniqueName: \"kubernetes.io/projected/0fe286da-6465-4d55-b062-0af4426ebf20-kube-api-access-t9lj6\") pod \"0fe286da-6465-4d55-b062-0af4426ebf20\" (UID: \"0fe286da-6465-4d55-b062-0af4426ebf20\") " Nov 01 20:46:00.021465 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.021444 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe286da-6465-4d55-b062-0af4426ebf20-plugins-config-volume" pod "0fe286da-6465-4d55-b062-0af4426ebf20" (UID: "0fe286da-6465-4d55-b062-0af4426ebf20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:00.021566 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.021548 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fe286da-6465-4d55-b062-0af4426ebf20-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:00.023370 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.023320 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe286da-6465-4d55-b062-0af4426ebf20-kube-api-access-t9lj6" pod "0fe286da-6465-4d55-b062-0af4426ebf20" (UID: "0fe286da-6465-4d55-b062-0af4426ebf20"). InnerVolumeSpecName "kube-api-access-t9lj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:00.122105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.122068 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a96f92-3084-4be6-b7a8-7f5f89c257e4-kube-api-access-g8mh9\" (UniqueName: \"kubernetes.io/projected/61a96f92-3084-4be6-b7a8-7f5f89c257e4-kube-api-access-g8mh9\") pod \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " Nov 01 20:46:00.122105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.122108 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a96f92-3084-4be6-b7a8-7f5f89c257e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a96f92-3084-4be6-b7a8-7f5f89c257e4-plugins-config-volume\") pod \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\" (UID: \"61a96f92-3084-4be6-b7a8-7f5f89c257e4\") " Nov 01 20:46:00.122593 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.122565 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a96f92-3084-4be6-b7a8-7f5f89c257e4-plugins-config-volume" pod "61a96f92-3084-4be6-b7a8-7f5f89c257e4" (UID: "61a96f92-3084-4be6-b7a8-7f5f89c257e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:00.122711 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.122671 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9lj6\" (UniqueName: \"kubernetes.io/projected/0fe286da-6465-4d55-b062-0af4426ebf20-kube-api-access-t9lj6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:00.124444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.124422 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a96f92-3084-4be6-b7a8-7f5f89c257e4-kube-api-access-g8mh9" pod "61a96f92-3084-4be6-b7a8-7f5f89c257e4" (UID: "61a96f92-3084-4be6-b7a8-7f5f89c257e4"). InnerVolumeSpecName "kube-api-access-g8mh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:00.223774 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.223749 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8mh9\" (UniqueName: \"kubernetes.io/projected/61a96f92-3084-4be6-b7a8-7f5f89c257e4-kube-api-access-g8mh9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:00.223896 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.223779 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a96f92-3084-4be6-b7a8-7f5f89c257e4-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:00.335176 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.335140 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn"] Nov 01 20:46:00.338742 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.338714 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc24d-bccf45fcf-6rlfn"] Nov 01 20:46:00.467049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.467017 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe286da-6465-4d55-b062-0af4426ebf20" path="/var/lib/kubelet/pods/0fe286da-6465-4d55-b062-0af4426ebf20/volumes" Nov 01 20:46:00.467539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.467525 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" path="/var/lib/kubelet/pods/41cd787f-33ca-49ad-968b-d86b755b5826/volumes" Nov 01 20:46:00.931324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.931236 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" Nov 01 20:46:00.931324 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.931236 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl" event={"ID":"61a96f92-3084-4be6-b7a8-7f5f89c257e4","Type":"ContainerDied","Data":"301a7a7322726c60158fbb9d7b1485e7f32d2ed39bd47096a52b94bb8d3474de"} Nov 01 20:46:00.931877 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:00.931404 5633 scope.go:122] "RemoveContainer" containerID="6c999acebff5dd60b2fe244ecd08180c1e3f9974e939231ae089e9f10364c26b" Nov 01 20:46:01.094908 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:01.094864 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl"] Nov 01 20:46:01.097185 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:01.097147 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:01.098882 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:01.098854 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc24d-55f4b9c8f5-66pwl"] Nov 01 20:46:01.471840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:01.471802 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:01.595042 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:01.595001 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:01.801433 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:01.801346 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:02.270811 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:02.270775 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:02.471372 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:02.471313 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" path="/var/lib/kubelet/pods/61a96f92-3084-4be6-b7a8-7f5f89c257e4/volumes" Nov 01 20:46:02.582666 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:02.582544 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:07.281284 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:07.281237 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:11.988592 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:11.988557 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:13.102556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:13.102520 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:46:16.156708 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:46:16.156671 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:46:16.688356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:16.688310 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:17.724606 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:17.724575 5633 scope.go:122] "RemoveContainer" containerID="d08721b374de749b15350aebe82452309647341cf77739b1302361de572688a8" Nov 01 20:46:17.736761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:17.736739 5633 scope.go:122] "RemoveContainer" containerID="cd2159612f313bad6b4599989a7f7ea20281ef387770169ce574711df889d9ca" Nov 01 20:46:21.375886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:21.375852 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:23.813523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:23.813484 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:23.931103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:23.931057 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:25.769919 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:25.769881 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:46:26.085920 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:26.085826 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:28.361584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:28.361543 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:30.786647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:30.786607 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:32.240463 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:32.240425 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:35.487197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:35.487163 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:40.201003 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:40.200960 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:42.323189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:42.323147 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:44.921694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:44.921658 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:46.157025 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:46:46.156985 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:46:49.584604 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:49.584559 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:54.283452 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:54.283366 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:46:59.052356 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:46:59.052289 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:03.691130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:03.691094 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:08.491800 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:08.491762 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:13.192964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:13.192915 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:16.157446 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:47:16.157409 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:47:17.955856 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:17.955802 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:22.582799 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:22.582754 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:27.297706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:27.297664 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:31.973323 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:31.973276 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:32.581130 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:32.581093 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:36.683627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:36.683586 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:41.372768 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:41.372719 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:46.063552 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:46.063508 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:46.159458 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:47:46.159421 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:47:50.752384 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:50.752312 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:47:55.410656 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:47:55.410616 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:00.089002 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:00.088959 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:04.809373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:04.809322 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:09.552864 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:09.552827 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:14.179443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:14.179399 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:16.156965 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:48:16.156923 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:48:18.896595 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:18.896557 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:23.572113 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:23.572027 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:24.812831 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:24.812784 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:24.993810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:24.993754 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:28.336728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:28.336682 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:29.048644 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:29.048609 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf"] Nov 01 20:48:29.097534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:29.097498 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3f541a-b8qnf"] Nov 01 20:48:29.588694 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:29.588659 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:29.761728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:29.761676 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:30.466654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:30.466621 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ace90c-e17a-4c7f-801c-b497f6aea9e8" path="/var/lib/kubelet/pods/90ace90c-e17a-4c7f-801c-b497f6aea9e8/volumes" Nov 01 20:48:32.918589 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:32.918551 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:33.090118 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:33.090076 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:34.130895 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:34.130857 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:48:37.884159 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:37.884124 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:41.046119 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:41.046086 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9"] Nov 01 20:48:41.095298 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:41.095262 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc1a334-mdgw9"] Nov 01 20:48:42.466082 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:42.466034 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5969d01-2f2e-4b0e-958a-8e1f325717cd" path="/var/lib/kubelet/pods/b5969d01-2f2e-4b0e-958a-8e1f325717cd/volumes" Nov 01 20:48:42.588938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:42.588898 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:46.158124 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:48:46.158077 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:48:46.968354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:46.968302 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:48:47.485979 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:47.485926 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:52.087586 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:52.087550 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:48:53.048179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:53.048134 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh"] Nov 01 20:48:53.097615 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:53.097582 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be971ce-cm5qh"] Nov 01 20:48:54.471764 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:54.471732 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8a8455-bc05-4921-a9ed-b6f1597f613a" path="/var/lib/kubelet/pods/7f8a8455-bc05-4921-a9ed-b6f1597f613a/volumes" Nov 01 20:48:56.787840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:48:56.787799 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:01.586631 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:01.586591 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:06.296535 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:06.296494 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:08.046151 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:08.046119 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5"] Nov 01 20:49:08.096505 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:08.096467 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33c3eb4b-5pxm5"] Nov 01 20:49:08.467285 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:08.467242 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8291468-b8de-4e33-9456-5dc06ab4f216" path="/var/lib/kubelet/pods/e8291468-b8de-4e33-9456-5dc06ab4f216/volumes" Nov 01 20:49:09.316933 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:09.316899 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:11.162103 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:11.162064 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:15.884884 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:15.884850 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:16.157129 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:49:16.157041 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:49:20.591983 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:20.591949 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:21.054843 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:21.054807 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn"] Nov 01 20:49:21.103974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:21.103944 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3234497e-bjxkn"] Nov 01 20:49:22.467018 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:22.466988 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31038fb4-07b6-413c-b139-05ca83384495" path="/var/lib/kubelet/pods/31038fb4-07b6-413c-b139-05ca83384495/volumes" Nov 01 20:49:25.298286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:25.298230 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:30.092653 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:30.092612 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:33.265139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:33.265100 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:34.051662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:34.051629 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw"] Nov 01 20:49:34.100424 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:34.100384 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3933d258-6vptw"] Nov 01 20:49:34.466316 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:34.466284 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e22317-6474-4ab7-87bf-e7b020abe7e9" path="/var/lib/kubelet/pods/60e22317-6474-4ab7-87bf-e7b020abe7e9/volumes" Nov 01 20:49:34.796162 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:34.796080 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:39.497175 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:39.497142 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:42.078797 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:42.078759 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2"] Nov 01 20:49:42.128109 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:42.128075 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21400cc4-fc7b2"] Nov 01 20:49:42.467660 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:42.467626 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1304efc-7d24-46d1-a2ca-c741105f7ff9" path="/var/lib/kubelet/pods/a1304efc-7d24-46d1-a2ca-c741105f7ff9/volumes" Nov 01 20:49:44.314197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:44.314158 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:46.156888 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:49:46.156852 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:49:49.061581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:49.061541 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:53.689290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:53.689200 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:57.047550 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:57.047513 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh"] Nov 01 20:49:57.096923 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:57.096886 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5c92d6-mnpmh"] Nov 01 20:49:58.411228 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:58.411189 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:49:58.467925 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:49:58.467886 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56286159-4951-47f7-9ed2-3aafde0b09de" path="/var/lib/kubelet/pods/56286159-4951-47f7-9ed2-3aafde0b09de/volumes" Nov 01 20:50:02.153818 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:02.153773 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:02.279601 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:02.279561 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:03.084072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:03.084036 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:07.792674 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:07.792637 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:12.490798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:12.490761 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:16.156374 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:50:16.156309 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:50:17.187562 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:17.187529 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:17.822049 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:17.822014 5633 scope.go:122] "RemoveContainer" containerID="92338358ceefe10944a8212377a7d8365b9c6fb051c7e2dd0112a14a0e3d2849" Nov 01 20:50:17.831556 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:17.831534 5633 scope.go:122] "RemoveContainer" containerID="7c52e269a8c02dca283e28210665e3412d5afa56cb6bd0e8fb2ecf23aba59184" Nov 01 20:50:17.840454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:17.840432 5633 scope.go:122] "RemoveContainer" containerID="bcb0698056b2932b6400d6027adff82b1ff12fe1462bfa3c6c0526b8e21b2e71" Nov 01 20:50:17.858146 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:17.858113 5633 scope.go:122] "RemoveContainer" containerID="e5461d91ee7f7f9dc5a09458313ac3b20b8c1bca344b2a4b56281aefb93755f7" Nov 01 20:50:21.892239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:21.892201 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:24.155940 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:24.155899 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s"] Nov 01 20:50:24.156534 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:24.156117 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" containerID="cri-o://51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f" gracePeriod=30 Nov 01 20:50:26.695746 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:26.695706 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:29.412699 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.412673 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:50:29.523249 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523214 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-credential-socket\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523267 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-istio-token\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-istio-token\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523308 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599e829d-0f1c-496e-bf40-f476c03bb904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e829d-0f1c-496e-bf40-f476c03bb904-istiod-ca-cert\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523421 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-data\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523485 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523451 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-kube-api-access-48jm6\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-kube-api-access-48jm6\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523527 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/599e829d-0f1c-496e-bf40-f476c03bb904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e829d-0f1c-496e-bf40-f476c03bb904-istio-podinfo\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523570 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-socket\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523581 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-credential-socket" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:29.523733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523636 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-certs\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.523733 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523677 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-envoy\") pod \"599e829d-0f1c-496e-bf40-f476c03bb904\" (UID: \"599e829d-0f1c-496e-bf40-f476c03bb904\") " Nov 01 20:50:29.524011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523773 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-data" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:29.524011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523881 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-socket" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:29.524011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523885 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599e829d-0f1c-496e-bf40-f476c03bb904-istiod-ca-cert" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:29.524011 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.523968 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-certs" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:29.524527 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.524500 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.524662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.524532 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.524662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.524548 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.524662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.524563 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.524662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.524578 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/599e829d-0f1c-496e-bf40-f476c03bb904-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.525903 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.525878 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/599e829d-0f1c-496e-bf40-f476c03bb904-istio-podinfo" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:50:29.526030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.525953 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-istio-token" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:29.526030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.525978 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-envoy" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:29.526387 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.526365 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-kube-api-access-48jm6" pod "599e829d-0f1c-496e-bf40-f476c03bb904" (UID: "599e829d-0f1c-496e-bf40-f476c03bb904"). InnerVolumeSpecName "kube-api-access-48jm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:29.595787 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.595747 5633 generic.go:356] "Generic (PLEG): container finished" podID="599e829d-0f1c-496e-bf40-f476c03bb904" containerID="51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f" exitCode=0 Nov 01 20:50:29.596032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.595841 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" event={"ID":"599e829d-0f1c-496e-bf40-f476c03bb904","Type":"ContainerDied","Data":"51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f"} Nov 01 20:50:29.596032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.595898 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" Nov 01 20:50:29.596032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.595937 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s" event={"ID":"599e829d-0f1c-496e-bf40-f476c03bb904","Type":"ContainerDied","Data":"66920d4fa253307147942ca99b172658a86858b6c475593e351d210e67106754"} Nov 01 20:50:29.596032 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.595966 5633 scope.go:122] "RemoveContainer" containerID="51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f" Nov 01 20:50:29.607217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.607194 5633 scope.go:122] "RemoveContainer" containerID="93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827" Nov 01 20:50:29.616909 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.616888 5633 scope.go:122] "RemoveContainer" containerID="51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f" Nov 01 20:50:29.617248 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:50:29.617226 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f\": container with ID starting with 51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f not found: ID does not exist" containerID="51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f" Nov 01 20:50:29.617354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.617257 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f"} err="failed to get container status \"51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f\": rpc error: code = NotFound desc = could not find container \"51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f\": container with ID starting with 51d9c03b4c67be9d94e9a320102722a2b9172c056556fb826515c0c944aa0c1f not found: ID does not exist" Nov 01 20:50:29.617354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.617278 5633 scope.go:122] "RemoveContainer" containerID="93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827" Nov 01 20:50:29.617578 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:50:29.617561 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827\": container with ID starting with 93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827 not found: ID does not exist" containerID="93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827" Nov 01 20:50:29.617648 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.617584 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827"} err="failed to get container status \"93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827\": rpc error: code = NotFound desc = could not find container \"93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827\": container with ID starting with 93503b81162eabd05d401e80be7892ed4d002c188406891796c5d955c1830827 not found: ID does not exist" Nov 01 20:50:29.625217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.625190 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/599e829d-0f1c-496e-bf40-f476c03bb904-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.625354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.625222 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.625354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.625234 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jm6\" (UniqueName: \"kubernetes.io/projected/599e829d-0f1c-496e-bf40-f476c03bb904-kube-api-access-48jm6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.625354 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.625244 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/599e829d-0f1c-496e-bf40-f476c03bb904-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:29.761379 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.761317 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s"] Nov 01 20:50:29.764046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:29.764025 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-h8w9s"] Nov 01 20:50:30.466514 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:30.466480 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" path="/var/lib/kubelet/pods/599e829d-0f1c-496e-bf40-f476c03bb904/volumes" Nov 01 20:50:31.391440 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:31.391405 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:33.605308 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:33.605270 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:36.084200 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:36.084154 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:39.291976 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.291938 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn"] Nov 01 20:50:39.292430 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.292176 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" containerID="cri-o://716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3" gracePeriod=30 Nov 01 20:50:39.341294 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.341254 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2"] Nov 01 20:50:39.341552 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.341529 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" containerID="cri-o://2a3aac4129ef88601146fde75c09918f439021d710f469ee3944a2e6badd57dc" gracePeriod=30 Nov 01 20:50:39.343994 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.343971 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc"] Nov 01 20:50:39.344197 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.344176 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" containerID="cri-o://1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2" gracePeriod=30 Nov 01 20:50:39.391581 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.391542 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4"] Nov 01 20:50:39.391816 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:39.391788 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" containerID="cri-o://14d8ada7a9f3ae4ed8f778fd7b7a99d4dcc97f4a2e1bfbf1fed5d931673654ff" gracePeriod=30 Nov 01 20:50:40.790632 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:40.790597 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:42.686741 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:42.686704 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfjk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:50:42.687149 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:42.686773 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:50:42.717046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:42.717009 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kjfvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:50:42.717209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:42.717107 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:50:43.810201 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:43.810161 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6xnbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:50:43.810709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:43.810222 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:50:43.993008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:43.992969 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lc5x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:50:43.993183 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:43.993022 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:50:44.612051 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.611737 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:50:44.662198 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.662163 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:50:44.697881 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.697848 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-kube-api-access-szv7g\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-kube-api-access-szv7g\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.697900 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-credential-socket\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.697972 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-socket\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698088 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.698009 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istiod-ca-cert\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.698110 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-envoy\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.698150 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-token\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-token\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.698185 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-data\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698279 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.698241 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-certs\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.698526 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.698287 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-podinfo\") pod \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\" (UID: \"f8f494f1-e9ae-427e-bae3-bf80db4fb48a\") " Nov 01 20:50:44.699209 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.699173 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-data" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.699401 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.699384 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-socket" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.699567 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.699551 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-credential-socket" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.701996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.701674 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-certs" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.701996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.701968 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istiod-ca-cert" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:44.701996 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.701966 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-podinfo" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:50:44.704886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.704858 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-kube-api-access-szv7g" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "kube-api-access-szv7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.705116 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.705095 5633 generic.go:356] "Generic (PLEG): container finished" podID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerID="14d8ada7a9f3ae4ed8f778fd7b7a99d4dcc97f4a2e1bfbf1fed5d931673654ff" exitCode=0 Nov 01 20:50:44.705192 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.705169 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" event={"ID":"a1ec2309-8a59-4003-a19b-9e54529e6ff6","Type":"ContainerDied","Data":"14d8ada7a9f3ae4ed8f778fd7b7a99d4dcc97f4a2e1bfbf1fed5d931673654ff"} Nov 01 20:50:44.706290 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.706269 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-token" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.707302 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.707280 5633 generic.go:356] "Generic (PLEG): container finished" podID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerID="2a3aac4129ef88601146fde75c09918f439021d710f469ee3944a2e6badd57dc" exitCode=0 Nov 01 20:50:44.707448 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.707408 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" event={"ID":"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e","Type":"ContainerDied","Data":"2a3aac4129ef88601146fde75c09918f439021d710f469ee3944a2e6badd57dc"} Nov 01 20:50:44.707618 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.707596 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-envoy" pod "f8f494f1-e9ae-427e-bae3-bf80db4fb48a" (UID: "f8f494f1-e9ae-427e-bae3-bf80db4fb48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.709007 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.708983 5633 generic.go:356] "Generic (PLEG): container finished" podID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerID="716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3" exitCode=0 Nov 01 20:50:44.709142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.709036 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" Nov 01 20:50:44.709142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.709067 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" event={"ID":"f8f494f1-e9ae-427e-bae3-bf80db4fb48a","Type":"ContainerDied","Data":"716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3"} Nov 01 20:50:44.709142 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.709118 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn" event={"ID":"f8f494f1-e9ae-427e-bae3-bf80db4fb48a","Type":"ContainerDied","Data":"7691ec880b8b2efc17cb9e79d707c04cb087c5d492491f96a6a279cb8179519d"} Nov 01 20:50:44.709305 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.709150 5633 scope.go:122] "RemoveContainer" containerID="716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3" Nov 01 20:50:44.710954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.710903 5633 generic.go:356] "Generic (PLEG): container finished" podID="631d8092-14c4-49b7-b045-94f8077fce9f" containerID="1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2" exitCode=0 Nov 01 20:50:44.710954 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.710925 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" event={"ID":"631d8092-14c4-49b7-b045-94f8077fce9f","Type":"ContainerDied","Data":"1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2"} Nov 01 20:50:44.711110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.710960 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" Nov 01 20:50:44.711110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.710981 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc" event={"ID":"631d8092-14c4-49b7-b045-94f8077fce9f","Type":"ContainerDied","Data":"b0be211369fc7f7c467ed0aee4636babd04c9363f74d5c50807fe4ace4aef940"} Nov 01 20:50:44.722368 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.722313 5633 scope.go:122] "RemoveContainer" containerID="716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3" Nov 01 20:50:44.722713 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:50:44.722695 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3\": container with ID starting with 716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3 not found: ID does not exist" containerID="716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3" Nov 01 20:50:44.722786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.722721 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3"} err="failed to get container status \"716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3\": rpc error: code = NotFound desc = could not find container \"716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3\": container with ID starting with 716bf0adfd0e7a5ce04075b35ae46d20c480fa837444da793cbecd07177f74a3 not found: ID does not exist" Nov 01 20:50:44.722786 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.722738 5633 scope.go:122] "RemoveContainer" containerID="1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2" Nov 01 20:50:44.734915 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.734892 5633 scope.go:122] "RemoveContainer" containerID="1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2" Nov 01 20:50:44.735211 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:50:44.735189 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2\": container with ID starting with 1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2 not found: ID does not exist" containerID="1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2" Nov 01 20:50:44.735282 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.735219 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2"} err="failed to get container status \"1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2\": rpc error: code = NotFound desc = could not find container \"1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2\": container with ID starting with 1fb1b53a1d53680193946b853156a32416849ec345e24677b4c8324aa4c16ed2 not found: ID does not exist" Nov 01 20:50:44.736724 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.736706 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:50:44.739035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.739015 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:50:44.799584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799543 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631d8092-14c4-49b7-b045-94f8077fce9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631d8092-14c4-49b7-b045-94f8077fce9f-istiod-ca-cert\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.799748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799631 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-socket\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.799748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799685 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-data\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.799748 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799728 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/631d8092-14c4-49b7-b045-94f8077fce9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631d8092-14c4-49b7-b045-94f8077fce9f-istio-podinfo\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.799890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799760 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-certs\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.799890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799799 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-token\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-token\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.799890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799827 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-envoy\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.799890 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799869 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-credential-socket\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.800072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799913 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-istio-token\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-istio-token\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.800072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799938 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istiod-ca-cert\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.800072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799977 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-certs\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.800072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.799981 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-socket" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.800072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800003 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631d8092-14c4-49b7-b045-94f8077fce9f-istiod-ca-cert" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:44.800072 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800010 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-credential-socket\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800079 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-kube-api-access-2ds4l\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-kube-api-access-2ds4l\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800126 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-data\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800221 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-envoy\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800248 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-kube-api-access-jf4br\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-kube-api-access-jf4br\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800233 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-credential-socket" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800278 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-socket\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800306 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-data" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800319 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-podinfo\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800365 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-certs\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800415 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-credential-socket\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800416 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istiod-ca-cert" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:44.800457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800441 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-data\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800477 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-token\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-token\") pod \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\" (UID: \"a1ec2309-8a59-4003-a19b-9e54529e6ff6\") " Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800504 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-socket\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800505 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-certs" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800542 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-envoy\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800626 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-podinfo\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800672 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istiod-ca-cert\") pod \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\" (UID: \"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e\") " Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800720 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-certs" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800727 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-kube-api-access-qbmdw\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-kube-api-access-qbmdw\") pod \"631d8092-14c4-49b7-b045-94f8077fce9f\" (UID: \"631d8092-14c4-49b7-b045-94f8077fce9f\") " Nov 01 20:50:44.801099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.800841 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-credential-socket" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801193 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801215 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631d8092-14c4-49b7-b045-94f8077fce9f-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801231 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801246 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801261 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801274 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801288 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801302 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801316 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801351 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801366 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801380 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801393 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801407 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801421 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szv7g\" (UniqueName: \"kubernetes.io/projected/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-kube-api-access-szv7g\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801437 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801450 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f494f1-e9ae-427e-bae3-bf80db4fb48a-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.801662 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.801663 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-data" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.803713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.803680 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-kube-api-access-2ds4l" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "kube-api-access-2ds4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.803852 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.803714 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-kube-api-access-qbmdw" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "kube-api-access-qbmdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.804111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.804081 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/631d8092-14c4-49b7-b045-94f8077fce9f-istio-podinfo" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:50:44.804357 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.804317 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-data" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.804901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.804553 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-token" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.804901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.804688 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-istio-token" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.805064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.804826 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-envoy" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.805064 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.804929 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-socket" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.805186 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.805117 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-certs" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.805230 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-socket" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.805362 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.805322 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-credential-socket" pod "631d8092-14c4-49b7-b045-94f8077fce9f" (UID: "631d8092-14c4-49b7-b045-94f8077fce9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.805524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.805485 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-envoy" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.805646 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.805576 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istiod-ca-cert" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:44.806422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.806391 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-podinfo" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:50:44.806539 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.806496 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-envoy" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:44.806948 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.806928 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-kube-api-access-jf4br" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "kube-api-access-jf4br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.807134 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.807119 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-token" pod "a1ec2309-8a59-4003-a19b-9e54529e6ff6" (UID: "a1ec2309-8a59-4003-a19b-9e54529e6ff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:44.807220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.807201 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-podinfo" pod "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" (UID: "7cd04fc0-d37e-4de9-8f6f-aa319f847b4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.902912 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.902944 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ds4l\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-kube-api-access-2ds4l\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.902954 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.902969 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.902978 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4br\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-kube-api-access-jf4br\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.902987 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.902995 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903003 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903001 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903011 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631d8092-14c4-49b7-b045-94f8077fce9f-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903019 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903027 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903038 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903049 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903057 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903065 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903073 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbmdw\" (UniqueName: \"kubernetes.io/projected/631d8092-14c4-49b7-b045-94f8077fce9f-kube-api-access-qbmdw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903081 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631d8092-14c4-49b7-b045-94f8077fce9f-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903089 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.903969 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.903097 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ec2309-8a59-4003-a19b-9e54529e6ff6-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:44.975879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.975844 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn"] Nov 01 20:50:44.978568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:44.978546 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6xnbn"] Nov 01 20:50:45.316714 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.316668 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc"] Nov 01 20:50:45.319458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.319429 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kjfvc"] Nov 01 20:50:45.482358 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.482313 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:45.718531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.718499 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" Nov 01 20:50:45.718531 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.718512 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2" event={"ID":"7cd04fc0-d37e-4de9-8f6f-aa319f847b4e","Type":"ContainerDied","Data":"50ab5a5fd80b814406407e80b4d7247facde199c509cde12c6a647beceb4aee5"} Nov 01 20:50:45.718803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.718598 5633 scope.go:122] "RemoveContainer" containerID="2a3aac4129ef88601146fde75c09918f439021d710f469ee3944a2e6badd57dc" Nov 01 20:50:45.722412 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.722393 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" Nov 01 20:50:45.722518 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.722379 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4" event={"ID":"a1ec2309-8a59-4003-a19b-9e54529e6ff6","Type":"ContainerDied","Data":"cb34f109390a1200117f115f2b5012eb1396bc0ba6a6b2184f28e1bb94256aba"} Nov 01 20:50:45.730080 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.730059 5633 scope.go:122] "RemoveContainer" containerID="14d8ada7a9f3ae4ed8f778fd7b7a99d4dcc97f4a2e1bfbf1fed5d931673654ff" Nov 01 20:50:45.882084 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.882032 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2"] Nov 01 20:50:45.884934 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.884908 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lc5x2"] Nov 01 20:50:45.994194 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.994099 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4"] Nov 01 20:50:45.996761 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:45.996741 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfjk4"] Nov 01 20:50:46.156462 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:50:46.156419 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:50:46.467622 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:46.467591 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" path="/var/lib/kubelet/pods/631d8092-14c4-49b7-b045-94f8077fce9f/volumes" Nov 01 20:50:46.468041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:46.468027 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" path="/var/lib/kubelet/pods/7cd04fc0-d37e-4de9-8f6f-aa319f847b4e/volumes" Nov 01 20:50:46.468432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:46.468420 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" path="/var/lib/kubelet/pods/a1ec2309-8a59-4003-a19b-9e54529e6ff6/volumes" Nov 01 20:50:46.469143 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:46.469130 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" path="/var/lib/kubelet/pods/f8f494f1-e9ae-427e-bae3-bf80db4fb48a/volumes" Nov 01 20:50:47.052952 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:47.052916 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64"] Nov 01 20:50:47.101945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:47.101908 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31bbeaf-csg64"] Nov 01 20:50:48.466788 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:48.466754 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f2597c-78ef-41a7-afd2-0686614f8925" path="/var/lib/kubelet/pods/f7f2597c-78ef-41a7-afd2-0686614f8925/volumes" Nov 01 20:50:50.180416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:50.180365 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:50.718963 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:50.718924 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:50.885267 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:50.885230 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:54.028713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.028674 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:54.197686 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.197652 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:54.805968 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.805932 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5"] Nov 01 20:50:54.806454 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806442 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806457 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806471 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806476 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806487 5633 cpu_manager.go:414] "containerMap: removing container" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806492 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806500 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806505 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" Nov 01 20:50:54.806508 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806510 5633 cpu_manager.go:414] "containerMap: removing container" podUID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" containerName="payload-pre-processing" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806515 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" containerName="payload-pre-processing" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806523 5633 cpu_manager.go:414] "containerMap: removing container" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806528 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806542 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806547 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806556 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0fe286da-6465-4d55-b062-0af4426ebf20" containerName="payload-processing" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806561 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe286da-6465-4d55-b062-0af4426ebf20" containerName="payload-processing" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806567 5633 cpu_manager.go:414] "containerMap: removing container" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806571 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806582 5633 cpu_manager.go:414] "containerMap: removing container" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806587 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cb4a4a-12c5-48b3-89af-85ce2713e530" containerName="cleanup" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806595 5633 cpu_manager.go:414] "containerMap: removing container" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806600 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806607 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f7f2597c-78ef-41a7-afd2-0686614f8925" containerName="revoke-keys" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806612 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f2597c-78ef-41a7-afd2-0686614f8925" containerName="revoke-keys" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806619 5633 cpu_manager.go:414] "containerMap: removing container" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806625 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806631 5633 cpu_manager.go:414] "containerMap: removing container" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806636 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806697 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88479da-823a-482b-8fb1-cb680c329d05" containerName="cleanup" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806709 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806714 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ec2309-8a59-4003-a19b-9e54529e6ff6" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806720 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a96f92-3084-4be6-b7a8-7f5f89c257e4" containerName="payload-pre-processing" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806727 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a00de5-a2dd-4a89-a10d-57993d24f88f" containerName="cleanup" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806733 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f494f1-e9ae-427e-bae3-bf80db4fb48a" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806738 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe286da-6465-4d55-b062-0af4426ebf20" containerName="payload-processing" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806744 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cd787f-33ca-49ad-968b-d86b755b5826" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806750 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806755 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f2597c-78ef-41a7-afd2-0686614f8925" containerName="revoke-keys" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806762 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="631d8092-14c4-49b7-b045-94f8077fce9f" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806767 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="599e829d-0f1c-496e-bf40-f476c03bb904" containerName="istio-proxy" Nov 01 20:50:54.806993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.806774 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd04fc0-d37e-4de9-8f6f-aa319f847b4e" containerName="istio-proxy" Nov 01 20:50:54.811841 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.811820 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:54.865382 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.865346 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5"] Nov 01 20:50:54.888239 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.888208 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 20:50:54.893079 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.893045 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:50:54.909650 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.909622 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:54.909798 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.909674 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:54.909874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.909786 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:54.909874 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.909855 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxcrc\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-kube-api-access-jxcrc\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:54.910772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.910745 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-1c1bf2169d38\"" Nov 01 20:50:54.910772 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.910764 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 20:50:54.910993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.910777 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 20:50:54.910993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.910777 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 20:50:54.910993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.910841 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"e2e-cfl-224e35-provider-cred\"" Nov 01 20:50:54.910993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.910914 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-433dc433b175\"" Nov 01 20:50:54.910993 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:54.910966 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-dhzm4\"" Nov 01 20:50:55.011307 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.011274 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.011549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.011400 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxcrc\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-kube-api-access-jxcrc\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.011549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.011462 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.011549 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.011505 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.012180 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.012150 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.012286 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.012192 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.014202 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.014183 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.068099 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.068014 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxcrc\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-kube-api-access-jxcrc\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.125030 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.124985 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.499469 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.499441 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5"] Nov 01 20:50:55.500153 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:50:55.500126 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fec82f9_a867_42b9_8804_a10555fc2234.slice/crio-a3e9cea2a9d9a9fd0b4fab9a59f33480a20601b0f2ccba7b8da5416ee9904582 WatchSource:0}: Error finding container a3e9cea2a9d9a9fd0b4fab9a59f33480a20601b0f2ccba7b8da5416ee9904582: Status 404 returned error can't find the container with id a3e9cea2a9d9a9fd0b4fab9a59f33480a20601b0f2ccba7b8da5416ee9904582 Nov 01 20:50:55.784602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.784511 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" event={"ID":"7fec82f9-a867-42b9-8804-a10555fc2234","Type":"ContainerStarted","Data":"86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d"} Nov 01 20:50:55.784602 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.784569 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" event={"ID":"7fec82f9-a867-42b9-8804-a10555fc2234","Type":"ContainerStarted","Data":"a3e9cea2a9d9a9fd0b4fab9a59f33480a20601b0f2ccba7b8da5416ee9904582"} Nov 01 20:50:55.784840 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.784629 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:50:55.901033 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:55.900957 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" podStartSLOduration=1.9009361070000002 podStartE2EDuration="1.900936107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:55.896369774 +0000 UTC m=+2863.570629030" watchObservedRunningTime="2026-11-01 20:50:55.900936107 +0000 UTC m=+2863.575195365" Nov 01 20:50:57.103497 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.103457 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:50:57.307257 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.307218 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5"] Nov 01 20:50:57.341964 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.341917 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz"] Nov 01 20:50:57.345598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.345576 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.396970 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.396928 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-0d210cf9f3e2\"" Nov 01 20:50:57.416233 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.416202 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz"] Nov 01 20:50:57.535745 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.535712 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.535926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.535766 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.535926 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.535884 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5b8\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-kube-api-access-2t5b8\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.536041 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.536008 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.637693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.637534 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.637693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.637589 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.637693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.637625 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.637693 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.637673 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5b8\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-kube-api-access-2t5b8\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.638557 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.638528 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.638681 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.638550 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.642674 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.642641 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.700879 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.700839 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5b8\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-kube-api-access-2t5b8\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:57.964246 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:57.964203 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:58.346114 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:58.346084 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz"] Nov 01 20:50:58.347929 ip-10-0-128-78 kubenswrapper[5633]: W1101 20:50:58.347899 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4beed60c_cfdc_4996_ac13_3e1e49dc3fcf.slice/crio-8eb976bc85a80e89ecbdaf25b08a7fb64b3056b0c1a1f494a4f4e60dd83ddc98 WatchSource:0}: Error finding container 8eb976bc85a80e89ecbdaf25b08a7fb64b3056b0c1a1f494a4f4e60dd83ddc98: Status 404 returned error can't find the container with id 8eb976bc85a80e89ecbdaf25b08a7fb64b3056b0c1a1f494a4f4e60dd83ddc98 Nov 01 20:50:58.806432 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:58.806394 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" event={"ID":"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf","Type":"ContainerStarted","Data":"a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d"} Nov 01 20:50:58.806627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:58.806460 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" event={"ID":"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf","Type":"ContainerStarted","Data":"8eb976bc85a80e89ecbdaf25b08a7fb64b3056b0c1a1f494a4f4e60dd83ddc98"} Nov 01 20:50:58.806627 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:58.806616 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:50:58.920016 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:58.919958 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" podStartSLOduration=1.919944819 podStartE2EDuration="1.919944819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:50:58.91831379 +0000 UTC m=+2866.592573044" watchObservedRunningTime="2026-11-01 20:50:58.919944819 +0000 UTC m=+2866.594204084" Nov 01 20:50:59.590838 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:50:59.590805 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:02.800444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:02.800396 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" podUID="7fec82f9-a867-42b9-8804-a10555fc2234" containerName="payload-processing" containerID="cri-o://86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d" gracePeriod=25 Nov 01 20:51:03.064062 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.063992 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:51:03.095974 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.095942 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-routing-overlay-volume\") pod \"7fec82f9-a867-42b9-8804-a10555fc2234\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " Nov 01 20:51:03.096199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.096027 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-kube-api-access-jxcrc\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-kube-api-access-jxcrc\") pod \"7fec82f9-a867-42b9-8804-a10555fc2234\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " Nov 01 20:51:03.096199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.096121 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-plugins-config-volume\") pod \"7fec82f9-a867-42b9-8804-a10555fc2234\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " Nov 01 20:51:03.096199 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.096195 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-provider-credentials\") pod \"7fec82f9-a867-42b9-8804-a10555fc2234\" (UID: \"7fec82f9-a867-42b9-8804-a10555fc2234\") " Nov 01 20:51:03.096455 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.096433 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-routing-overlay-volume" pod "7fec82f9-a867-42b9-8804-a10555fc2234" (UID: "7fec82f9-a867-42b9-8804-a10555fc2234"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:03.096548 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.096524 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-plugins-config-volume" pod "7fec82f9-a867-42b9-8804-a10555fc2234" (UID: "7fec82f9-a867-42b9-8804-a10555fc2234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:03.096596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.096545 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:03.098443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.098402 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-kube-api-access-jxcrc" pod "7fec82f9-a867-42b9-8804-a10555fc2234" (UID: "7fec82f9-a867-42b9-8804-a10555fc2234"). InnerVolumeSpecName "kube-api-access-jxcrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:03.098443 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.098433 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-provider-credentials" pod "7fec82f9-a867-42b9-8804-a10555fc2234" (UID: "7fec82f9-a867-42b9-8804-a10555fc2234"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:03.197444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.197409 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxcrc\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-kube-api-access-jxcrc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:03.197444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.197440 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fec82f9-a867-42b9-8804-a10555fc2234-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:03.197444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.197449 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fec82f9-a867-42b9-8804-a10555fc2234-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:03.837111 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.837074 5633 generic.go:356] "Generic (PLEG): container finished" podID="7fec82f9-a867-42b9-8804-a10555fc2234" containerID="86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d" exitCode=0 Nov 01 20:51:03.837664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.837127 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" Nov 01 20:51:03.837664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.837157 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" event={"ID":"7fec82f9-a867-42b9-8804-a10555fc2234","Type":"ContainerDied","Data":"86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d"} Nov 01 20:51:03.837664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.837215 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5" event={"ID":"7fec82f9-a867-42b9-8804-a10555fc2234","Type":"ContainerDied","Data":"a3e9cea2a9d9a9fd0b4fab9a59f33480a20601b0f2ccba7b8da5416ee9904582"} Nov 01 20:51:03.837664 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.837237 5633 scope.go:122] "RemoveContainer" containerID="86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d" Nov 01 20:51:03.850296 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.850275 5633 scope.go:122] "RemoveContainer" containerID="86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d" Nov 01 20:51:03.850682 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:51:03.850662 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d\": container with ID starting with 86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d not found: ID does not exist" containerID="86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d" Nov 01 20:51:03.850750 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:03.850690 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d"} err="failed to get container status \"86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d\": rpc error: code = NotFound desc = could not find container \"86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d\": container with ID starting with 86e764ec4d1380a9a10c2a10a8412bc4f7e9906a95f3427d6d480cc0b142b74d not found: ID does not exist" Nov 01 20:51:04.000217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:04.000174 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5"] Nov 01 20:51:04.048345 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:04.048297 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845f7slk5"] Nov 01 20:51:04.301373 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:04.301322 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:04.466462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:04.466426 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fec82f9-a867-42b9-8804-a10555fc2234" path="/var/lib/kubelet/pods/7fec82f9-a867-42b9-8804-a10555fc2234/volumes" Nov 01 20:51:08.989110 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:08.989054 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:09.815444 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:09.815413 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:51:13.708870 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:13.708827 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:16.156374 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:51:16.156309 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:51:16.769652 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:16.769611 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:51:17.899095 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:17.899057 5633 scope.go:122] "RemoveContainer" containerID="45579878cba531449c08d6eb55671ebe411ab8793e8ab7c28f5a69499ccced99" Nov 01 20:51:17.908569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:17.908548 5633 scope.go:122] "RemoveContainer" containerID="5e9609e9c80e2aec4fa857495b0c3649d4a8ec261a3340a600eb90c48c55f692" Nov 01 20:51:17.917450 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:17.917405 5633 scope.go:122] "RemoveContainer" containerID="6a9146cdbcf84afcc7508a0593ce65888d3ede2686204a238cd48c7132553040" Nov 01 20:51:17.926697 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:17.926679 5633 scope.go:122] "RemoveContainer" containerID="0aca7c6998df22d268c5dc8e8e069231b0c7b2d36f9bf1a10f653196878a1660" Nov 01 20:51:18.377559 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:18.377522 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:23.088967 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:23.088874 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:27.782108 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:27.782071 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:29.114457 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:29.114421 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:51:32.494520 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:32.494483 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:33.944217 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:33.944177 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:37.192801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:37.192765 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:41.875056 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:41.875021 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:46.157368 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:51:46.157311 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:51:46.592719 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:46.592683 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:51.374887 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:51.374847 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:51:56.086901 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:51:56.086860 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:00.780771 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:00.780728 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:05.515715 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:05.515668 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:10.180303 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:10.180245 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:14.883538 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:14.883496 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:16.156733 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:52:16.156692 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:52:17.985598 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:17.985570 5633 scope.go:122] "RemoveContainer" containerID="14222b65d710fd15fbf06069e1a078cb6db78f9819fe89040087263ceba5409b" Nov 01 20:52:19.586803 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:19.586766 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:24.255569 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:24.255530 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:29.031462 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:29.031424 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:33.692642 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:33.692605 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:34.278672 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:34.278632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:38.388094 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:38.388048 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:43.082596 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:43.082538 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:46.156260 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:52:46.156223 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:52:47.838989 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:47.838946 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:49.441263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:49.441223 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:49.560722 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:49.560684 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:52.491726 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:52.491646 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:52:57.238000 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:52:57.237958 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:01.883266 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:01.883230 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:06.595584 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:06.595544 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:11.348647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:11.348605 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:15.992204 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:15.992168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:16.156769 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:53:16.156731 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:53:20.734542 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:20.734490 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:25.392709 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:25.392673 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:30.137026 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:30.136990 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:32.565914 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:32.565872 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:32.728888 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:32.728848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:34.612046 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:34.612007 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:34.776422 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:34.776372 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:37.202278 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:37.202241 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:53:39.488220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:39.488185 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:44.193965 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:44.193915 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:46.155753 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:53:46.155718 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:53:48.206638 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:48.206599 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 20:53:48.881612 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:48.881574 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz"] Nov 01 20:53:48.908220 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:48.908168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:53.584945 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:53.584910 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:53:53.882248 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:53.882147 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" podUID="4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" containerName="payload-processing" containerID="cri-o://a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d" gracePeriod=25 Nov 01 20:53:54.140107 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.140045 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:53:54.272008 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.271974 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-plugins-config-volume\") pod \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " Nov 01 20:53:54.272212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.272034 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-provider-credentials\") pod \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " Nov 01 20:53:54.272212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.272102 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-routing-overlay-volume\") pod \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " Nov 01 20:53:54.272212 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.272149 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-kube-api-access-2t5b8\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-kube-api-access-2t5b8\") pod \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\" (UID: \"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf\") " Nov 01 20:53:54.272486 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.272459 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-plugins-config-volume" pod "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" (UID: "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:53:54.272574 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.272513 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-routing-overlay-volume" pod "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" (UID: "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:53:54.272645 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.272626 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:54.272700 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.272650 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:54.274577 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.274548 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-provider-credentials" pod "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" (UID: "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:53:54.274713 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.274614 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-kube-api-access-2t5b8" pod "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" (UID: "4beed60c-cfdc-4996-ac13-3e1e49dc3fcf"). InnerVolumeSpecName "kube-api-access-2t5b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:53:54.373189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.373149 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5b8\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-kube-api-access-2t5b8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:54.373189 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.373182 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:53:54.828756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.828656 5633 generic.go:356] "Generic (PLEG): container finished" podID="4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" containerID="a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d" exitCode=0 Nov 01 20:53:54.828756 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.828731 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" Nov 01 20:53:54.829375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.828781 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" event={"ID":"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf","Type":"ContainerDied","Data":"a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d"} Nov 01 20:53:54.829375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.828857 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz" event={"ID":"4beed60c-cfdc-4996-ac13-3e1e49dc3fcf","Type":"ContainerDied","Data":"8eb976bc85a80e89ecbdaf25b08a7fb64b3056b0c1a1f494a4f4e60dd83ddc98"} Nov 01 20:53:54.829375 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.828890 5633 scope.go:122] "RemoveContainer" containerID="a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d" Nov 01 20:53:54.838434 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.838413 5633 scope.go:122] "RemoveContainer" containerID="a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d" Nov 01 20:53:54.838730 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:53:54.838711 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d\": container with ID starting with a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d not found: ID does not exist" containerID="a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d" Nov 01 20:53:54.838810 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.838740 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d"} err="failed to get container status \"a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d\": rpc error: code = NotFound desc = could not find container \"a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d\": container with ID starting with a5cecc2879b609ffb812bf0ef99bcff37f8d8f46b90cf76e885ed8e0e233217d not found: ID does not exist" Nov 01 20:53:54.991102 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:54.991064 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz"] Nov 01 20:53:55.040878 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:55.040846 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cbjxtcz"] Nov 01 20:53:56.465869 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:56.465835 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" path="/var/lib/kubelet/pods/4beed60c-cfdc-4996-ac13-3e1e49dc3fcf/volumes" Nov 01 20:53:58.286188 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:53:58.286151 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:03.017779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:03.017738 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:07.684098 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:07.684062 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:12.379541 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:12.379497 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:16.156790 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:54:16.156744 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:54:17.089383 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:17.089316 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:21.815263 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:21.815158 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:26.477779 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:26.477745 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:31.177406 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:31.177357 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:34.963830 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:34.963796 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:35.956688 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:35.956648 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:40.631560 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:40.631524 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:45.290524 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:45.290480 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:46.158398 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:54:46.158358 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:54:49.989023 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:49.988982 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:54.682184 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:54.682140 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:54:59.386801 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:54:59.386759 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:05.430027 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:05.429988 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:08.796921 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:08.796878 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:13.490796 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:13.490752 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:16.156199 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:55:16.156161 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:55:18.238163 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:18.238122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:22.886599 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:22.886563 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:27.588809 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:27.588769 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:32.287179 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:32.287144 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:35.305416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:35.305369 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:36.981689 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:36.981653 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:41.688682 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:41.688632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:44.070287 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:44.070247 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:46.158274 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:55:46.158237 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:55:46.385876 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:46.385829 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:51.093360 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:51.093239 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:55:55.806717 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:55:55.806676 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:00.513886 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:00.513848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:05.199105 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:05.199065 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:09.896178 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:09.896138 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:14.602971 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:14.602931 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:16.156317 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:56:16.156278 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:56:19.457847 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:19.457810 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:24.074568 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:24.074529 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:28.780999 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:28.780956 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:33.479523 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:33.479483 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:35.639066 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:35.639029 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:38.178410 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:38.178366 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:42.901938 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:42.901903 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:46.156313 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:56:46.156278 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:56:47.584858 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:47.584803 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:52.288842 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:52.288808 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:56:56.985035 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:56:56.984999 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:01.688624 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:01.688592 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:06.367872 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:06.367829 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:10.908676 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:10.908634 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:11.080675 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:11.080614 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:15.681825 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:15.681783 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:16.156151 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:57:16.156107 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:57:20.102687 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:20.102629 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:20.219791 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:20.219748 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:20.369216 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:20.369118 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:25.059464 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:25.059371 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:29.762850 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:29.762809 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:34.498071 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:34.498028 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:35.987932 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:35.987888 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:39.194075 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:39.194039 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:43.973461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:43.973422 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:46.156801 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:57:46.156762 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:57:48.682626 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:48.682586 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:53.389728 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:53.389680 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:57:58.090139 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:57:58.090074 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:02.782355 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:02.782287 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:07.495458 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:07.495418 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:12.178517 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:12.178483 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:16.156430 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:58:16.156378 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:58:16.876515 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:16.876476 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:21.577706 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:21.577666 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:26.280171 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:26.280130 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:31.026752 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:31.026709 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:35.687313 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:35.687275 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:36.346654 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:36.346606 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:40.389065 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:40.389025 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:45.084158 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:45.084119 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:46.155914 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:58:46.155879 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:58:49.792770 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:49.792736 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:54.499301 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:54.499266 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:58:59.179623 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:58:59.179582 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:03.978461 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:03.978419 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:08.726778 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:08.726744 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:13.417647 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:13.417606 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:16.156142 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:59:16.156105 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:59:18.086164 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:18.086124 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:18.793349 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:18.793291 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:22.804742 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:22.804704 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:27.488416 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:27.488367 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:32.181817 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:32.181780 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:36.665439 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:36.665406 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:36.889213 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:36.889179 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:41.612630 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:41.612591 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:46.155886 ip-10-0-128-78 kubenswrapper[5633]: E1101 20:59:46.155839 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 20:59:46.285165 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:46.285130 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:50.986147 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:50.986106 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 20:59:55.676900 ip-10-0-128-78 kubenswrapper[5633]: I1101 20:59:55.676854 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:00.226971 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.226934 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj"] Nov 01 21:00:00.227507 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.227433 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7fec82f9-a867-42b9-8804-a10555fc2234" containerName="payload-processing" Nov 01 21:00:00.227507 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.227446 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fec82f9-a867-42b9-8804-a10555fc2234" containerName="payload-processing" Nov 01 21:00:00.227507 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.227463 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" containerName="payload-processing" Nov 01 21:00:00.227507 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.227469 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" containerName="payload-processing" Nov 01 21:00:00.227733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.227576 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fec82f9-a867-42b9-8804-a10555fc2234" containerName="payload-processing" Nov 01 21:00:00.227733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.227588 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4beed60c-cfdc-4996-ac13-3e1e49dc3fcf" containerName="payload-processing" Nov 01 21:00:00.231109 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.231087 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn"] Nov 01 21:00:00.231263 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.231225 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.234787 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.234769 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:00.283431 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.283402 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:00:00.283431 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.283421 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:00:00.333494 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.333463 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:00:00.339094 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.339051 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj"] Nov 01 21:00:00.342748 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.342713 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn"] Nov 01 21:00:00.392602 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.392566 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:00.407083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.407042 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45327f79-dd26-4db2-b190-27627e4f0cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.407083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.407091 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df06440-6270-4806-90a9-5a8dce074e42-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-7prtn\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:00.407315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.407271 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt297\" (UniqueName: \"kubernetes.io/projected/6df06440-6270-4806-90a9-5a8dce074e42-kube-api-access-bt297\") pod \"maas-api-key-cleanup-29892780-7prtn\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:00.407396 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.407363 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mr56\" (UniqueName: \"kubernetes.io/projected/45327f79-dd26-4db2-b190-27627e4f0cac-kube-api-access-4mr56\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.508791 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.508709 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt297\" (UniqueName: \"kubernetes.io/projected/6df06440-6270-4806-90a9-5a8dce074e42-kube-api-access-bt297\") pod \"maas-api-key-cleanup-29892780-7prtn\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:00.508791 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.508768 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mr56\" (UniqueName: \"kubernetes.io/projected/45327f79-dd26-4db2-b190-27627e4f0cac-kube-api-access-4mr56\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.509018 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.508796 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45327f79-dd26-4db2-b190-27627e4f0cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.509018 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.508815 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df06440-6270-4806-90a9-5a8dce074e42-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-7prtn\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:00.509475 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.509457 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df06440-6270-4806-90a9-5a8dce074e42-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-7prtn\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:00.509538 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.509517 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45327f79-dd26-4db2-b190-27627e4f0cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.564892 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.564855 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mr56\" (UniqueName: \"kubernetes.io/projected/45327f79-dd26-4db2-b190-27627e4f0cac-kube-api-access-4mr56\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.565039 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.564911 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt297\" (UniqueName: \"kubernetes.io/projected/6df06440-6270-4806-90a9-5a8dce074e42-kube-api-access-bt297\") pod \"maas-api-key-cleanup-29892780-7prtn\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:00.846043 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.845949 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:00.852410 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:00.852389 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:01.222028 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:01.221998 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj"] Nov 01 21:00:01.222639 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:00:01.222605 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45327f79_dd26_4db2_b190_27627e4f0cac.slice/crio-f949fac9c0d0858c9237f5bd81c347113b0f4162518ecd2d3e81d938b1d7e853 WatchSource:0}: Error finding container f949fac9c0d0858c9237f5bd81c347113b0f4162518ecd2d3e81d938b1d7e853: Status 404 returned error can't find the container with id f949fac9c0d0858c9237f5bd81c347113b0f4162518ecd2d3e81d938b1d7e853 Nov 01 21:00:01.271160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:01.271135 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn"] Nov 01 21:00:01.273242 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:00:01.273212 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df06440_6270_4806_90a9_5a8dce074e42.slice/crio-891d2b3f2f7a755df3bdda2107bc4fbb578059face324e06fa232d356cc459a5 WatchSource:0}: Error finding container 891d2b3f2f7a755df3bdda2107bc4fbb578059face324e06fa232d356cc459a5: Status 404 returned error can't find the container with id 891d2b3f2f7a755df3bdda2107bc4fbb578059face324e06fa232d356cc459a5 Nov 01 21:00:01.842300 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:01.842205 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" event={"ID":"6df06440-6270-4806-90a9-5a8dce074e42","Type":"ContainerStarted","Data":"dbd0726ec9410bf2437ac02445b8c360419fdd38dbd46683ff6078fd2f4b255a"} Nov 01 21:00:01.842300 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:01.842270 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" event={"ID":"6df06440-6270-4806-90a9-5a8dce074e42","Type":"ContainerStarted","Data":"891d2b3f2f7a755df3bdda2107bc4fbb578059face324e06fa232d356cc459a5"} Nov 01 21:00:01.843755 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:01.843725 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" event={"ID":"45327f79-dd26-4db2-b190-27627e4f0cac","Type":"ContainerStarted","Data":"cc9d18029ce9aa0b4871f00114b7915d2ad0ed3b5c96680819ea034969d8d198"} Nov 01 21:00:01.843895 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:01.843777 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" event={"ID":"45327f79-dd26-4db2-b190-27627e4f0cac","Type":"ContainerStarted","Data":"f949fac9c0d0858c9237f5bd81c347113b0f4162518ecd2d3e81d938b1d7e853"} Nov 01 21:00:01.955577 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:01.955517 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" podStartSLOduration=1.9555029579999998 podStartE2EDuration="1.955502958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:01.953783101 +0000 UTC m=+3409.628042357" watchObservedRunningTime="2026-11-01 21:00:01.955502958 +0000 UTC m=+3409.629762214" Nov 01 21:00:02.067072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:02.067015 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" podStartSLOduration=2.066999396 podStartE2EDuration="2.066999396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:02.065618103 +0000 UTC m=+3409.739877361" watchObservedRunningTime="2026-11-01 21:00:02.066999396 +0000 UTC m=+3409.741258652" Nov 01 21:00:02.851235 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:02.851199 5633 generic.go:356] "Generic (PLEG): container finished" podID="45327f79-dd26-4db2-b190-27627e4f0cac" containerID="cc9d18029ce9aa0b4871f00114b7915d2ad0ed3b5c96680819ea034969d8d198" exitCode=60 Nov 01 21:00:02.851795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:02.851290 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" event={"ID":"45327f79-dd26-4db2-b190-27627e4f0cac","Type":"ContainerDied","Data":"cc9d18029ce9aa0b4871f00114b7915d2ad0ed3b5c96680819ea034969d8d198"} Nov 01 21:00:02.851795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:02.851557 5633 scope.go:122] "RemoveContainer" containerID="cc9d18029ce9aa0b4871f00114b7915d2ad0ed3b5c96680819ea034969d8d198" Nov 01 21:00:02.853250 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:02.853230 5633 generic.go:356] "Generic (PLEG): container finished" podID="6df06440-6270-4806-90a9-5a8dce074e42" containerID="dbd0726ec9410bf2437ac02445b8c360419fdd38dbd46683ff6078fd2f4b255a" exitCode=60 Nov 01 21:00:02.853358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:02.853288 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" event={"ID":"6df06440-6270-4806-90a9-5a8dce074e42","Type":"ContainerDied","Data":"dbd0726ec9410bf2437ac02445b8c360419fdd38dbd46683ff6078fd2f4b255a"} Nov 01 21:00:02.853534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:02.853520 5633 scope.go:122] "RemoveContainer" containerID="dbd0726ec9410bf2437ac02445b8c360419fdd38dbd46683ff6078fd2f4b255a" Nov 01 21:00:03.861106 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.861074 5633 generic.go:356] "Generic (PLEG): container finished" podID="6df06440-6270-4806-90a9-5a8dce074e42" containerID="53134ee29180b3bf6bb3e604c7d24c8a434c6cc3b0584482dd3cc9e6b9d6c523" exitCode=60 Nov 01 21:00:03.861689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.861164 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" event={"ID":"6df06440-6270-4806-90a9-5a8dce074e42","Type":"ContainerDied","Data":"53134ee29180b3bf6bb3e604c7d24c8a434c6cc3b0584482dd3cc9e6b9d6c523"} Nov 01 21:00:03.861689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.861225 5633 scope.go:122] "RemoveContainer" containerID="dbd0726ec9410bf2437ac02445b8c360419fdd38dbd46683ff6078fd2f4b255a" Nov 01 21:00:03.861689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.861518 5633 scope.go:122] "RemoveContainer" containerID="53134ee29180b3bf6bb3e604c7d24c8a434c6cc3b0584482dd3cc9e6b9d6c523" Nov 01 21:00:03.861842 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:03.861764 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-7prtn_odh-ai-gateway-infra(6df06440-6270-4806-90a9-5a8dce074e42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" podUID="6df06440-6270-4806-90a9-5a8dce074e42" Nov 01 21:00:03.862976 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.862946 5633 generic.go:356] "Generic (PLEG): container finished" podID="45327f79-dd26-4db2-b190-27627e4f0cac" containerID="faf933f55dea0b46c8de5cc5f0094e8e08b907bcc8092b6029dd860d60040169" exitCode=60 Nov 01 21:00:03.863078 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.862999 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" event={"ID":"45327f79-dd26-4db2-b190-27627e4f0cac","Type":"ContainerDied","Data":"faf933f55dea0b46c8de5cc5f0094e8e08b907bcc8092b6029dd860d60040169"} Nov 01 21:00:03.863246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.863232 5633 scope.go:122] "RemoveContainer" containerID="faf933f55dea0b46c8de5cc5f0094e8e08b907bcc8092b6029dd860d60040169" Nov 01 21:00:03.863492 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:03.863475 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj_odh-ai-gateway-infra(45327f79-dd26-4db2-b190-27627e4f0cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" Nov 01 21:00:03.898540 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:03.898518 5633 scope.go:122] "RemoveContainer" containerID="cc9d18029ce9aa0b4871f00114b7915d2ad0ed3b5c96680819ea034969d8d198" Nov 01 21:00:04.871753 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:04.871726 5633 scope.go:122] "RemoveContainer" containerID="53134ee29180b3bf6bb3e604c7d24c8a434c6cc3b0584482dd3cc9e6b9d6c523" Nov 01 21:00:04.872141 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:04.871969 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-7prtn_odh-ai-gateway-infra(6df06440-6270-4806-90a9-5a8dce074e42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" podUID="6df06440-6270-4806-90a9-5a8dce074e42" Nov 01 21:00:04.873907 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:04.873883 5633 scope.go:122] "RemoveContainer" containerID="faf933f55dea0b46c8de5cc5f0094e8e08b907bcc8092b6029dd860d60040169" Nov 01 21:00:04.874179 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:04.874156 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj_odh-ai-gateway-infra(45327f79-dd26-4db2-b190-27627e4f0cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" Nov 01 21:00:05.101465 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:05.101424 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:07.802944 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.802906 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj"] Nov 01 21:00:07.808369 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.808318 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.862624 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.862591 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj"] Nov 01 21:00:07.888853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.888820 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.889060 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.888864 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxkt\" (UniqueName: \"kubernetes.io/projected/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-kube-api-access-bnxkt\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.889060 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.888980 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.903100 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.903066 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:00:07.990400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.990365 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.990609 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.990420 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxkt\" (UniqueName: \"kubernetes.io/projected/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-kube-api-access-bnxkt\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.990609 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.990473 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.991087 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.991065 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:07.992821 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:07.992795 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:08.048957 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:08.048923 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxkt\" (UniqueName: \"kubernetes.io/projected/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-kube-api-access-bnxkt\") pod \"maas-api-revoke-sub-95d44c61-bcrvj\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:08.121645 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:08.121562 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:08.493092 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:08.493067 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj"] Nov 01 21:00:08.494616 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:00:08.494585 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76a9d29_6cb7_4d7c_b903_9f56ae84b91d.slice/crio-04eefc08f76be6b73f75d48bac50aed3567aa450c9b31c231c35fd51c65da208 WatchSource:0}: Error finding container 04eefc08f76be6b73f75d48bac50aed3567aa450c9b31c231c35fd51c65da208: Status 404 returned error can't find the container with id 04eefc08f76be6b73f75d48bac50aed3567aa450c9b31c231c35fd51c65da208 Nov 01 21:00:08.898661 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:08.898620 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" event={"ID":"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d","Type":"ContainerStarted","Data":"7620f068bd8eadaf412c092ea0dec98696682ab22aab4b4d7b2b9b0bf0816c8a"} Nov 01 21:00:08.899094 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:08.898684 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" event={"ID":"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d","Type":"ContainerStarted","Data":"04eefc08f76be6b73f75d48bac50aed3567aa450c9b31c231c35fd51c65da208"} Nov 01 21:00:09.010859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:09.010805 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" podStartSLOduration=2.010789183 podStartE2EDuration="2.010789183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:09.00978643 +0000 UTC m=+3416.684045686" watchObservedRunningTime="2026-11-01 21:00:09.010789183 +0000 UTC m=+3416.685048438" Nov 01 21:00:09.790744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:09.790701 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:09.905345 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:09.905303 5633 generic.go:356] "Generic (PLEG): container finished" podID="e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" containerID="7620f068bd8eadaf412c092ea0dec98696682ab22aab4b4d7b2b9b0bf0816c8a" exitCode=0 Nov 01 21:00:09.905758 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:09.905390 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" event={"ID":"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d","Type":"ContainerDied","Data":"7620f068bd8eadaf412c092ea0dec98696682ab22aab4b4d7b2b9b0bf0816c8a"} Nov 01 21:00:11.047776 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.047742 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:11.122307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.122277 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-openshift-ca\") pod \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " Nov 01 21:00:11.122522 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.122388 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-xks-ca\") pod \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " Nov 01 21:00:11.122522 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.122436 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-kube-api-access-bnxkt\" (UniqueName: \"kubernetes.io/projected/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-kube-api-access-bnxkt\") pod \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\" (UID: \"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d\") " Nov 01 21:00:11.122745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.122703 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-openshift-ca" pod "e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" (UID: "e76a9d29-6cb7-4d7c-b903-9f56ae84b91d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:11.122903 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.122840 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:11.124706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.124679 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-xks-ca" pod "e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" (UID: "e76a9d29-6cb7-4d7c-b903-9f56ae84b91d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:00:11.124807 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.124719 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-kube-api-access-bnxkt" pod "e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" (UID: "e76a9d29-6cb7-4d7c-b903-9f56ae84b91d"). InnerVolumeSpecName "kube-api-access-bnxkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:11.224160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.224120 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:11.224160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.224154 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxkt\" (UniqueName: \"kubernetes.io/projected/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d-kube-api-access-bnxkt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:11.917493 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.917453 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" Nov 01 21:00:11.917493 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.917476 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj" event={"ID":"e76a9d29-6cb7-4d7c-b903-9f56ae84b91d","Type":"ContainerDied","Data":"04eefc08f76be6b73f75d48bac50aed3567aa450c9b31c231c35fd51c65da208"} Nov 01 21:00:11.917729 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:11.917556 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04eefc08f76be6b73f75d48bac50aed3567aa450c9b31c231c35fd51c65da208" Nov 01 21:00:14.523800 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:14.523763 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:16.156792 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:16.156751 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:00:16.460628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:16.460547 5633 scope.go:122] "RemoveContainer" containerID="53134ee29180b3bf6bb3e604c7d24c8a434c6cc3b0584482dd3cc9e6b9d6c523" Nov 01 21:00:16.462041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:16.461024 5633 scope.go:122] "RemoveContainer" containerID="faf933f55dea0b46c8de5cc5f0094e8e08b907bcc8092b6029dd860d60040169" Nov 01 21:00:16.946561 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:16.946521 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" event={"ID":"6df06440-6270-4806-90a9-5a8dce074e42","Type":"ContainerStarted","Data":"351c40aa90aed7ef9c9297fe563a8d98da0cdac3897bc02681e2857da104d741"} Nov 01 21:00:16.948553 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:16.948517 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" event={"ID":"45327f79-dd26-4db2-b190-27627e4f0cac","Type":"ContainerStarted","Data":"d46c52f761f10cdbd0be03e7306b0cae78f446d474a31ddeb7b66a04d59dd19c"} Nov 01 21:00:17.956905 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.956869 5633 generic.go:356] "Generic (PLEG): container finished" podID="6df06440-6270-4806-90a9-5a8dce074e42" containerID="351c40aa90aed7ef9c9297fe563a8d98da0cdac3897bc02681e2857da104d741" exitCode=60 Nov 01 21:00:17.957438 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.956941 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" event={"ID":"6df06440-6270-4806-90a9-5a8dce074e42","Type":"ContainerDied","Data":"351c40aa90aed7ef9c9297fe563a8d98da0cdac3897bc02681e2857da104d741"} Nov 01 21:00:17.957438 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.957001 5633 scope.go:122] "RemoveContainer" containerID="53134ee29180b3bf6bb3e604c7d24c8a434c6cc3b0584482dd3cc9e6b9d6c523" Nov 01 21:00:17.957438 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.957391 5633 scope.go:122] "RemoveContainer" containerID="351c40aa90aed7ef9c9297fe563a8d98da0cdac3897bc02681e2857da104d741" Nov 01 21:00:17.957688 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:17.957667 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-7prtn_odh-ai-gateway-infra(6df06440-6270-4806-90a9-5a8dce074e42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" podUID="6df06440-6270-4806-90a9-5a8dce074e42" Nov 01 21:00:17.959089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.959069 5633 generic.go:356] "Generic (PLEG): container finished" podID="45327f79-dd26-4db2-b190-27627e4f0cac" containerID="d46c52f761f10cdbd0be03e7306b0cae78f446d474a31ddeb7b66a04d59dd19c" exitCode=60 Nov 01 21:00:17.959176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.959097 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" event={"ID":"45327f79-dd26-4db2-b190-27627e4f0cac","Type":"ContainerDied","Data":"d46c52f761f10cdbd0be03e7306b0cae78f446d474a31ddeb7b66a04d59dd19c"} Nov 01 21:00:17.959432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.959418 5633 scope.go:122] "RemoveContainer" containerID="d46c52f761f10cdbd0be03e7306b0cae78f446d474a31ddeb7b66a04d59dd19c" Nov 01 21:00:17.959656 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:17.959640 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj_odh-ai-gateway-infra(45327f79-dd26-4db2-b190-27627e4f0cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" Nov 01 21:00:17.969494 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:17.969470 5633 scope.go:122] "RemoveContainer" containerID="faf933f55dea0b46c8de5cc5f0094e8e08b907bcc8092b6029dd860d60040169" Nov 01 21:00:18.069733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:18.069698 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn"] Nov 01 21:00:18.180156 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:18.180122 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj"] Nov 01 21:00:19.112624 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.112593 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:19.147200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.147176 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:19.194569 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.194526 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:19.209437 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.209394 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df06440-6270-4806-90a9-5a8dce074e42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df06440-6270-4806-90a9-5a8dce074e42-maas-api-service-ca\") pod \"6df06440-6270-4806-90a9-5a8dce074e42\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " Nov 01 21:00:19.209604 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.209458 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45327f79-dd26-4db2-b190-27627e4f0cac-kube-api-access-4mr56\" (UniqueName: \"kubernetes.io/projected/45327f79-dd26-4db2-b190-27627e4f0cac-kube-api-access-4mr56\") pod \"45327f79-dd26-4db2-b190-27627e4f0cac\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " Nov 01 21:00:19.209604 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.209556 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df06440-6270-4806-90a9-5a8dce074e42-kube-api-access-bt297\" (UniqueName: \"kubernetes.io/projected/6df06440-6270-4806-90a9-5a8dce074e42-kube-api-access-bt297\") pod \"6df06440-6270-4806-90a9-5a8dce074e42\" (UID: \"6df06440-6270-4806-90a9-5a8dce074e42\") " Nov 01 21:00:19.209716 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.209629 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45327f79-dd26-4db2-b190-27627e4f0cac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45327f79-dd26-4db2-b190-27627e4f0cac-maas-api-service-ca\") pod \"45327f79-dd26-4db2-b190-27627e4f0cac\" (UID: \"45327f79-dd26-4db2-b190-27627e4f0cac\") " Nov 01 21:00:19.209949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.209923 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df06440-6270-4806-90a9-5a8dce074e42-maas-api-service-ca" pod "6df06440-6270-4806-90a9-5a8dce074e42" (UID: "6df06440-6270-4806-90a9-5a8dce074e42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:19.210006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.209973 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45327f79-dd26-4db2-b190-27627e4f0cac-maas-api-service-ca" pod "45327f79-dd26-4db2-b190-27627e4f0cac" (UID: "45327f79-dd26-4db2-b190-27627e4f0cac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:19.211853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.211832 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45327f79-dd26-4db2-b190-27627e4f0cac-kube-api-access-4mr56" pod "45327f79-dd26-4db2-b190-27627e4f0cac" (UID: "45327f79-dd26-4db2-b190-27627e4f0cac"). InnerVolumeSpecName "kube-api-access-4mr56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:19.211923 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.211851 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df06440-6270-4806-90a9-5a8dce074e42-kube-api-access-bt297" pod "6df06440-6270-4806-90a9-5a8dce074e42" (UID: "6df06440-6270-4806-90a9-5a8dce074e42"). InnerVolumeSpecName "kube-api-access-bt297". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:19.310894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.310853 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45327f79-dd26-4db2-b190-27627e4f0cac-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:19.310894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.310895 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df06440-6270-4806-90a9-5a8dce074e42-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:19.311105 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.310911 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mr56\" (UniqueName: \"kubernetes.io/projected/45327f79-dd26-4db2-b190-27627e4f0cac-kube-api-access-4mr56\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:19.311105 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.310927 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt297\" (UniqueName: \"kubernetes.io/projected/6df06440-6270-4806-90a9-5a8dce074e42-kube-api-access-bt297\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:19.976032 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.975999 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" Nov 01 21:00:19.976032 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.976015 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn" event={"ID":"6df06440-6270-4806-90a9-5a8dce074e42","Type":"ContainerDied","Data":"891d2b3f2f7a755df3bdda2107bc4fbb578059face324e06fa232d356cc459a5"} Nov 01 21:00:19.976318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.976091 5633 scope.go:122] "RemoveContainer" containerID="351c40aa90aed7ef9c9297fe563a8d98da0cdac3897bc02681e2857da104d741" Nov 01 21:00:19.978400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.978315 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" Nov 01 21:00:19.978400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.978354 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj" event={"ID":"45327f79-dd26-4db2-b190-27627e4f0cac","Type":"ContainerDied","Data":"f949fac9c0d0858c9237f5bd81c347113b0f4162518ecd2d3e81d938b1d7e853"} Nov 01 21:00:19.988079 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:19.988050 5633 scope.go:122] "RemoveContainer" containerID="d46c52f761f10cdbd0be03e7306b0cae78f446d474a31ddeb7b66a04d59dd19c" Nov 01 21:00:20.139419 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.139255 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn"] Nov 01 21:00:20.141962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.141938 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-7prtn"] Nov 01 21:00:20.295691 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.295579 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj"] Nov 01 21:00:20.302391 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.302353 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892780-tmsjj"] Nov 01 21:00:20.357556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.357514 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:20.466899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.466859 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" path="/var/lib/kubelet/pods/45327f79-dd26-4db2-b190-27627e4f0cac/volumes" Nov 01 21:00:20.467286 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.467272 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df06440-6270-4806-90a9-5a8dce074e42" path="/var/lib/kubelet/pods/6df06440-6270-4806-90a9-5a8dce074e42/volumes" Nov 01 21:00:20.524288 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:20.524253 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:21.116994 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.116910 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6"] Nov 01 21:00:21.117440 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117425 5633 cpu_manager.go:414] "containerMap: removing container" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117442 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117455 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" containerName="revoke-keys" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117460 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" containerName="revoke-keys" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117468 5633 cpu_manager.go:414] "containerMap: removing container" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117473 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117479 5633 cpu_manager.go:414] "containerMap: removing container" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117483 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117490 5633 cpu_manager.go:414] "containerMap: removing container" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:21.117495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117493 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:21.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117583 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:21.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117593 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:21.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117599 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" containerName="revoke-keys" Nov 01 21:00:21.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117604 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:21.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117611 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:21.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117615 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:21.117767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.117621 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:21.122253 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.122235 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.175828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.175792 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6"] Nov 01 21:00:21.221687 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.221647 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:00:21.221687 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.221680 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:00:21.221917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.221653 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:00:21.223262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.223245 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:00:21.228829 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.228792 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc119687-432b-464e-8694-67981bd29159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.228927 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.228860 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zrf\" (UniqueName: \"kubernetes.io/projected/cc119687-432b-464e-8694-67981bd29159-kube-api-access-g8zrf\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.228927 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.228912 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc119687-432b-464e-8694-67981bd29159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.330073 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.330031 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc119687-432b-464e-8694-67981bd29159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.330262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.330083 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zrf\" (UniqueName: \"kubernetes.io/projected/cc119687-432b-464e-8694-67981bd29159-kube-api-access-g8zrf\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.330262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.330103 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc119687-432b-464e-8694-67981bd29159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.330808 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.330786 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc119687-432b-464e-8694-67981bd29159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.332596 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.332576 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc119687-432b-464e-8694-67981bd29159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.386509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.386428 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zrf\" (UniqueName: \"kubernetes.io/projected/cc119687-432b-464e-8694-67981bd29159-kube-api-access-g8zrf\") pod \"maas-api-revoke-sub-0f116018-ncvg6\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.438545 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.438504 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:21.813977 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.813947 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6"] Nov 01 21:00:21.815260 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:00:21.815228 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc119687_432b_464e_8694_67981bd29159.slice/crio-c16a21e893c78711390012069b0e79e0ea5fc6b146431e39d825a9f895fcd1be WatchSource:0}: Error finding container c16a21e893c78711390012069b0e79e0ea5fc6b146431e39d825a9f895fcd1be: Status 404 returned error can't find the container with id c16a21e893c78711390012069b0e79e0ea5fc6b146431e39d825a9f895fcd1be Nov 01 21:00:21.993702 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:21.993671 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" event={"ID":"cc119687-432b-464e-8694-67981bd29159","Type":"ContainerStarted","Data":"c16a21e893c78711390012069b0e79e0ea5fc6b146431e39d825a9f895fcd1be"} Nov 01 21:00:23.001436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:23.001395 5633 generic.go:356] "Generic (PLEG): container finished" podID="cc119687-432b-464e-8694-67981bd29159" containerID="37a8861eeddc8a736995f30aed5c4bcff19cbecac228578b5f5acead17573ec7" exitCode=0 Nov 01 21:00:23.001954 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:23.001487 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" event={"ID":"cc119687-432b-464e-8694-67981bd29159","Type":"ContainerDied","Data":"37a8861eeddc8a736995f30aed5c4bcff19cbecac228578b5f5acead17573ec7"} Nov 01 21:00:23.109775 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:23.109732 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:00:23.883151 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:23.883118 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:24.150085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.150058 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:24.258891 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.258859 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc119687-432b-464e-8694-67981bd29159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc119687-432b-464e-8694-67981bd29159-maas-api-openshift-ca\") pod \"cc119687-432b-464e-8694-67981bd29159\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " Nov 01 21:00:24.259071 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.258928 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc119687-432b-464e-8694-67981bd29159-kube-api-access-g8zrf\" (UniqueName: \"kubernetes.io/projected/cc119687-432b-464e-8694-67981bd29159-kube-api-access-g8zrf\") pod \"cc119687-432b-464e-8694-67981bd29159\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " Nov 01 21:00:24.259071 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.258994 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc119687-432b-464e-8694-67981bd29159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc119687-432b-464e-8694-67981bd29159-maas-api-xks-ca\") pod \"cc119687-432b-464e-8694-67981bd29159\" (UID: \"cc119687-432b-464e-8694-67981bd29159\") " Nov 01 21:00:24.259248 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.259226 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc119687-432b-464e-8694-67981bd29159-maas-api-openshift-ca" pod "cc119687-432b-464e-8694-67981bd29159" (UID: "cc119687-432b-464e-8694-67981bd29159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:24.261401 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.261370 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc119687-432b-464e-8694-67981bd29159-maas-api-xks-ca" pod "cc119687-432b-464e-8694-67981bd29159" (UID: "cc119687-432b-464e-8694-67981bd29159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:00:24.262056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.262028 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc119687-432b-464e-8694-67981bd29159-kube-api-access-g8zrf" pod "cc119687-432b-464e-8694-67981bd29159" (UID: "cc119687-432b-464e-8694-67981bd29159"). InnerVolumeSpecName "kube-api-access-g8zrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:24.360454 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.360407 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc119687-432b-464e-8694-67981bd29159-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:24.360454 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.360444 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8zrf\" (UniqueName: \"kubernetes.io/projected/cc119687-432b-464e-8694-67981bd29159-kube-api-access-g8zrf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:24.360454 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:24.360460 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc119687-432b-464e-8694-67981bd29159-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:25.014706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:25.014677 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" Nov 01 21:00:25.014906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:25.014710 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6" event={"ID":"cc119687-432b-464e-8694-67981bd29159","Type":"ContainerDied","Data":"c16a21e893c78711390012069b0e79e0ea5fc6b146431e39d825a9f895fcd1be"} Nov 01 21:00:25.014906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:25.014766 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c16a21e893c78711390012069b0e79e0ea5fc6b146431e39d825a9f895fcd1be" Nov 01 21:00:28.594781 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:28.594741 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:30.202877 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:30.202836 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:32.772700 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.772653 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w"] Nov 01 21:00:32.773115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.773102 5633 cpu_manager.go:414] "containerMap: removing container" podUID="cc119687-432b-464e-8694-67981bd29159" containerName="revoke-keys" Nov 01 21:00:32.773115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.773113 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc119687-432b-464e-8694-67981bd29159" containerName="revoke-keys" Nov 01 21:00:32.773186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.773150 5633 cpu_manager.go:414] "containerMap: removing container" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:32.773186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.773156 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df06440-6270-4806-90a9-5a8dce074e42" containerName="cleanup" Nov 01 21:00:32.773186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.773164 5633 cpu_manager.go:414] "containerMap: removing container" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:32.773186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.773169 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45327f79-dd26-4db2-b190-27627e4f0cac" containerName="cleanup" Nov 01 21:00:32.773315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.773237 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc119687-432b-464e-8694-67981bd29159" containerName="revoke-keys" Nov 01 21:00:32.776665 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.776647 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:32.834798 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.834757 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w"] Nov 01 21:00:32.879837 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.879799 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:00:32.879837 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.879802 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:00:32.879837 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.879812 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:00:32.880140 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.879881 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:00:32.948906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.948866 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:32.949085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.948986 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggjq\" (UniqueName: \"kubernetes.io/projected/f0d3f245-50b9-4509-bd6e-a313a8b563ca-kube-api-access-5ggjq\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:32.949126 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:32.949106 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.050609 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.050522 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.050781 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.050631 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.050781 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.050694 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggjq\" (UniqueName: \"kubernetes.io/projected/f0d3f245-50b9-4509-bd6e-a313a8b563ca-kube-api-access-5ggjq\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.051176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.051153 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.053128 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.053110 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.106860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.106821 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggjq\" (UniqueName: \"kubernetes.io/projected/f0d3f245-50b9-4509-bd6e-a313a8b563ca-kube-api-access-5ggjq\") pod \"maas-api-revoke-sub-2d395028-2gc2w\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.321112 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.321016 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:33.390146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.390098 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:33.762898 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:33.762868 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w"] Nov 01 21:00:33.763989 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:00:33.763955 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d3f245_50b9_4509_bd6e_a313a8b563ca.slice/crio-ed2cd10c5c7c1fa50eac515714ca4bf2829608d8e74292574f378aed2764cfdb WatchSource:0}: Error finding container ed2cd10c5c7c1fa50eac515714ca4bf2829608d8e74292574f378aed2764cfdb: Status 404 returned error can't find the container with id ed2cd10c5c7c1fa50eac515714ca4bf2829608d8e74292574f378aed2764cfdb Nov 01 21:00:34.070176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:34.070092 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" event={"ID":"f0d3f245-50b9-4509-bd6e-a313a8b563ca","Type":"ContainerStarted","Data":"ed2cd10c5c7c1fa50eac515714ca4bf2829608d8e74292574f378aed2764cfdb"} Nov 01 21:00:35.077727 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:35.077690 5633 generic.go:356] "Generic (PLEG): container finished" podID="f0d3f245-50b9-4509-bd6e-a313a8b563ca" containerID="b96f6543e4fd068a6e0db92fe01ef1e4bcaec380f1a7f0d92ceec3c63d8136fc" exitCode=0 Nov 01 21:00:35.078194 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:35.077785 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" event={"ID":"f0d3f245-50b9-4509-bd6e-a313a8b563ca","Type":"ContainerDied","Data":"b96f6543e4fd068a6e0db92fe01ef1e4bcaec380f1a7f0d92ceec3c63d8136fc"} Nov 01 21:00:35.856749 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:35.856709 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:00:36.227798 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.227768 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:36.386137 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.386103 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-openshift-ca\") pod \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " Nov 01 21:00:36.386405 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.386220 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-xks-ca\") pod \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " Nov 01 21:00:36.386405 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.386392 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d3f245-50b9-4509-bd6e-a313a8b563ca-kube-api-access-5ggjq\" (UniqueName: \"kubernetes.io/projected/f0d3f245-50b9-4509-bd6e-a313a8b563ca-kube-api-access-5ggjq\") pod \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\" (UID: \"f0d3f245-50b9-4509-bd6e-a313a8b563ca\") " Nov 01 21:00:36.386932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.386904 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-openshift-ca" pod "f0d3f245-50b9-4509-bd6e-a313a8b563ca" (UID: "f0d3f245-50b9-4509-bd6e-a313a8b563ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:36.388777 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.388748 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-xks-ca" pod "f0d3f245-50b9-4509-bd6e-a313a8b563ca" (UID: "f0d3f245-50b9-4509-bd6e-a313a8b563ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:00:36.388950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.388930 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d3f245-50b9-4509-bd6e-a313a8b563ca-kube-api-access-5ggjq" pod "f0d3f245-50b9-4509-bd6e-a313a8b563ca" (UID: "f0d3f245-50b9-4509-bd6e-a313a8b563ca"). InnerVolumeSpecName "kube-api-access-5ggjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:36.488068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.488036 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggjq\" (UniqueName: \"kubernetes.io/projected/f0d3f245-50b9-4509-bd6e-a313a8b563ca-kube-api-access-5ggjq\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:36.488068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.488064 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:36.488068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:36.488075 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d3f245-50b9-4509-bd6e-a313a8b563ca-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:37.008136 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:37.008099 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:37.090551 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:37.090511 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" event={"ID":"f0d3f245-50b9-4509-bd6e-a313a8b563ca","Type":"ContainerDied","Data":"ed2cd10c5c7c1fa50eac515714ca4bf2829608d8e74292574f378aed2764cfdb"} Nov 01 21:00:37.090736 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:37.090560 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed2cd10c5c7c1fa50eac515714ca4bf2829608d8e74292574f378aed2764cfdb" Nov 01 21:00:37.090736 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:37.090523 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w" Nov 01 21:00:37.990859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:37.990818 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:42.759735 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:42.759693 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:44.871704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:44.871675 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:44.995495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:44.995457 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:46.157412 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:00:46.157369 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:00:47.222003 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:47.221958 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:47.394143 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:47.394099 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:47.516498 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:47.516402 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:48.004846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.004810 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg"] Nov 01 21:00:48.005277 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.005264 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f0d3f245-50b9-4509-bd6e-a313a8b563ca" containerName="revoke-keys" Nov 01 21:00:48.005324 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.005279 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d3f245-50b9-4509-bd6e-a313a8b563ca" containerName="revoke-keys" Nov 01 21:00:48.005392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.005383 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d3f245-50b9-4509-bd6e-a313a8b563ca" containerName="revoke-keys" Nov 01 21:00:48.008715 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.008690 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.064684 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.064651 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:00:48.064684 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.064651 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:00:48.103355 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.103292 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sg44\" (UniqueName: \"kubernetes.io/projected/347fe460-132d-4582-a1bd-4ab20aa03e47-kube-api-access-6sg44\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.103536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.103436 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.103536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.103484 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.114182 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.114150 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:00:48.114457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.114181 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:00:48.117224 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.117197 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg"] Nov 01 21:00:48.204043 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.204003 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sg44\" (UniqueName: \"kubernetes.io/projected/347fe460-132d-4582-a1bd-4ab20aa03e47-kube-api-access-6sg44\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.204265 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.204075 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.204265 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.204108 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.204884 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.204861 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.206652 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.206632 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.260558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.260480 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sg44\" (UniqueName: \"kubernetes.io/projected/347fe460-132d-4582-a1bd-4ab20aa03e47-kube-api-access-6sg44\") pod \"maas-api-revoke-sub-fe2dfdd7-wxzvg\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.321832 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.321792 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:48.700712 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.700685 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg"] Nov 01 21:00:48.702093 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:00:48.702065 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347fe460_132d_4582_a1bd_4ab20aa03e47.slice/crio-3e9966fef7dacc053fddb15d75d70e23479bb7db9e150f3f52a2098429ff5cac WatchSource:0}: Error finding container 3e9966fef7dacc053fddb15d75d70e23479bb7db9e150f3f52a2098429ff5cac: Status 404 returned error can't find the container with id 3e9966fef7dacc053fddb15d75d70e23479bb7db9e150f3f52a2098429ff5cac Nov 01 21:00:48.876568 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:48.876525 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:00:49.161217 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:49.161178 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" event={"ID":"347fe460-132d-4582-a1bd-4ab20aa03e47","Type":"ContainerStarted","Data":"9a9da119ec88375fd33c7c8c7a16583cdb266070f39485fac70db1e2e710799c"} Nov 01 21:00:49.161435 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:49.161246 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" event={"ID":"347fe460-132d-4582-a1bd-4ab20aa03e47","Type":"ContainerStarted","Data":"3e9966fef7dacc053fddb15d75d70e23479bb7db9e150f3f52a2098429ff5cac"} Nov 01 21:00:49.273827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:49.273723 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" podStartSLOduration=2.273706907 podStartE2EDuration="2.273706907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:49.271993629 +0000 UTC m=+3456.946252900" watchObservedRunningTime="2026-11-01 21:00:49.273706907 +0000 UTC m=+3456.947966162" Nov 01 21:00:50.173371 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:50.173245 5633 generic.go:356] "Generic (PLEG): container finished" podID="347fe460-132d-4582-a1bd-4ab20aa03e47" containerID="9a9da119ec88375fd33c7c8c7a16583cdb266070f39485fac70db1e2e710799c" exitCode=0 Nov 01 21:00:50.173660 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:50.173291 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" event={"ID":"347fe460-132d-4582-a1bd-4ab20aa03e47","Type":"ContainerDied","Data":"9a9da119ec88375fd33c7c8c7a16583cdb266070f39485fac70db1e2e710799c"} Nov 01 21:00:51.316863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.316834 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:51.439547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.439466 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-openshift-ca\") pod \"347fe460-132d-4582-a1bd-4ab20aa03e47\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " Nov 01 21:00:51.439733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.439616 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347fe460-132d-4582-a1bd-4ab20aa03e47-kube-api-access-6sg44\" (UniqueName: \"kubernetes.io/projected/347fe460-132d-4582-a1bd-4ab20aa03e47-kube-api-access-6sg44\") pod \"347fe460-132d-4582-a1bd-4ab20aa03e47\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " Nov 01 21:00:51.439733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.439653 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-xks-ca\") pod \"347fe460-132d-4582-a1bd-4ab20aa03e47\" (UID: \"347fe460-132d-4582-a1bd-4ab20aa03e47\") " Nov 01 21:00:51.440254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.440220 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-openshift-ca" pod "347fe460-132d-4582-a1bd-4ab20aa03e47" (UID: "347fe460-132d-4582-a1bd-4ab20aa03e47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:51.442118 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.442087 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-xks-ca" pod "347fe460-132d-4582-a1bd-4ab20aa03e47" (UID: "347fe460-132d-4582-a1bd-4ab20aa03e47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:00:51.442544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.442431 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347fe460-132d-4582-a1bd-4ab20aa03e47-kube-api-access-6sg44" pod "347fe460-132d-4582-a1bd-4ab20aa03e47" (UID: "347fe460-132d-4582-a1bd-4ab20aa03e47"). InnerVolumeSpecName "kube-api-access-6sg44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:51.541102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.541057 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:51.541102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.541093 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sg44\" (UniqueName: \"kubernetes.io/projected/347fe460-132d-4582-a1bd-4ab20aa03e47-kube-api-access-6sg44\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:51.541102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:51.541103 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347fe460-132d-4582-a1bd-4ab20aa03e47-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:52.094826 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:52.094786 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:00:52.187045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:52.187009 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" event={"ID":"347fe460-132d-4582-a1bd-4ab20aa03e47","Type":"ContainerDied","Data":"3e9966fef7dacc053fddb15d75d70e23479bb7db9e150f3f52a2098429ff5cac"} Nov 01 21:00:52.187045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:52.187045 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg" Nov 01 21:00:52.187269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:52.187064 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9966fef7dacc053fddb15d75d70e23479bb7db9e150f3f52a2098429ff5cac" Nov 01 21:00:56.804554 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:00:56.804515 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:01.262527 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:01.262478 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:01:01.306457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:01.306408 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:01.472945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:01.472895 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:01.595289 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:01.595125 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:02.072073 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.072034 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l"] Nov 01 21:01:02.072523 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.072509 5633 cpu_manager.go:414] "containerMap: removing container" podUID="347fe460-132d-4582-a1bd-4ab20aa03e47" containerName="revoke-keys" Nov 01 21:01:02.072609 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.072524 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347fe460-132d-4582-a1bd-4ab20aa03e47" containerName="revoke-keys" Nov 01 21:01:02.072654 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.072642 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="347fe460-132d-4582-a1bd-4ab20aa03e47" containerName="revoke-keys" Nov 01 21:01:02.076924 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.076906 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.129233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.129201 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:01:02.129233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.129239 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:01:02.132004 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.131964 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l"] Nov 01 21:01:02.177599 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.177565 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:01:02.177837 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.177608 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:01:02.251641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.251608 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.251828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.251655 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.251828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.251672 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxt9\" (UniqueName: \"kubernetes.io/projected/d691955f-3007-4c7e-b6e9-5ebc64610484-kube-api-access-cwxt9\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.353268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.353164 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.353268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.353228 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.353268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.353253 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxt9\" (UniqueName: \"kubernetes.io/projected/d691955f-3007-4c7e-b6e9-5ebc64610484-kube-api-access-cwxt9\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.353851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.353832 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.355880 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.355857 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.410361 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.410289 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxt9\" (UniqueName: \"kubernetes.io/projected/d691955f-3007-4c7e-b6e9-5ebc64610484-kube-api-access-cwxt9\") pod \"maas-api-revoke-sub-3ac7d3ee-twr9l\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:02.689884 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:02.689796 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:03.064198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:03.064169 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l"] Nov 01 21:01:03.065640 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:01:03.065609 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd691955f_3007_4c7e_b6e9_5ebc64610484.slice/crio-c9130067f492555de8855c97edae2722ecefa7df2591d90d4e3a6d845b56ad9b WatchSource:0}: Error finding container c9130067f492555de8855c97edae2722ecefa7df2591d90d4e3a6d845b56ad9b: Status 404 returned error can't find the container with id c9130067f492555de8855c97edae2722ecefa7df2591d90d4e3a6d845b56ad9b Nov 01 21:01:03.254941 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:03.254897 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" event={"ID":"d691955f-3007-4c7e-b6e9-5ebc64610484","Type":"ContainerStarted","Data":"c9130067f492555de8855c97edae2722ecefa7df2591d90d4e3a6d845b56ad9b"} Nov 01 21:01:04.262503 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:04.262463 5633 generic.go:356] "Generic (PLEG): container finished" podID="d691955f-3007-4c7e-b6e9-5ebc64610484" containerID="b7eb754c868eea17247a326bf2898c587748c258fdbcde91e25ece676e6c2ba4" exitCode=0 Nov 01 21:01:04.262917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:04.262545 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" event={"ID":"d691955f-3007-4c7e-b6e9-5ebc64610484","Type":"ContainerDied","Data":"b7eb754c868eea17247a326bf2898c587748c258fdbcde91e25ece676e6c2ba4"} Nov 01 21:01:05.414970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.414943 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:05.588942 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.588843 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-openshift-ca\") pod \"d691955f-3007-4c7e-b6e9-5ebc64610484\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " Nov 01 21:01:05.588942 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.588882 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-xks-ca\") pod \"d691955f-3007-4c7e-b6e9-5ebc64610484\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " Nov 01 21:01:05.589284 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.588969 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d691955f-3007-4c7e-b6e9-5ebc64610484-kube-api-access-cwxt9\" (UniqueName: \"kubernetes.io/projected/d691955f-3007-4c7e-b6e9-5ebc64610484-kube-api-access-cwxt9\") pod \"d691955f-3007-4c7e-b6e9-5ebc64610484\" (UID: \"d691955f-3007-4c7e-b6e9-5ebc64610484\") " Nov 01 21:01:05.589349 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.589293 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-openshift-ca" pod "d691955f-3007-4c7e-b6e9-5ebc64610484" (UID: "d691955f-3007-4c7e-b6e9-5ebc64610484"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:01:05.591224 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.591192 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-xks-ca" pod "d691955f-3007-4c7e-b6e9-5ebc64610484" (UID: "d691955f-3007-4c7e-b6e9-5ebc64610484"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:01:05.591396 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.591325 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d691955f-3007-4c7e-b6e9-5ebc64610484-kube-api-access-cwxt9" pod "d691955f-3007-4c7e-b6e9-5ebc64610484" (UID: "d691955f-3007-4c7e-b6e9-5ebc64610484"). InnerVolumeSpecName "kube-api-access-cwxt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:01:05.690848 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.690809 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:05.690848 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.690842 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691955f-3007-4c7e-b6e9-5ebc64610484-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:05.690848 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:05.690852 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwxt9\" (UniqueName: \"kubernetes.io/projected/d691955f-3007-4c7e-b6e9-5ebc64610484-kube-api-access-cwxt9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:06.175230 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:06.175190 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:06.276649 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:06.276621 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" Nov 01 21:01:06.276848 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:06.276619 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l" event={"ID":"d691955f-3007-4c7e-b6e9-5ebc64610484","Type":"ContainerDied","Data":"c9130067f492555de8855c97edae2722ecefa7df2591d90d4e3a6d845b56ad9b"} Nov 01 21:01:06.276848 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:06.276754 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9130067f492555de8855c97edae2722ecefa7df2591d90d4e3a6d845b56ad9b" Nov 01 21:01:10.873017 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:10.872979 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:12.754133 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:12.754094 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:01:14.337915 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:14.337873 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:14.508090 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:14.508050 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:14.860617 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:14.860577 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:14.981161 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:14.981125 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:15.103114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:15.103068 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:15.599749 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:15.599713 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:16.021255 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:16.021214 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:16.156977 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:01:16.156940 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:01:16.189855 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:16.189817 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:16.689041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:16.688989 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:20.290865 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:20.290816 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:22.423528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:22.423488 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:22.552186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:22.552139 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:22.677874 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:22.677783 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:24.981389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:24.981347 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:26.572591 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:26.572557 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:01:28.267141 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:28.267102 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:28.402899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:28.402852 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:29.342384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:29.342321 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:29.469551 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:29.469517 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:29.686933 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:29.686853 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:33.312298 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:33.312261 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:34.378946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:34.378911 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:37.346201 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:37.346168 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:39.079523 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:39.079485 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:40.064805 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:40.064764 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:01:43.794201 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:43.794160 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:46.155866 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:01:46.155827 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:01:48.494910 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:48.494877 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:53.183654 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:53.183562 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:53.494640 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:53.494598 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:01:57.981120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:57.981079 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:01:59.339396 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.339351 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq"] Nov 01 21:01:59.339945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.339831 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d691955f-3007-4c7e-b6e9-5ebc64610484" containerName="revoke-keys" Nov 01 21:01:59.339945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.339845 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d691955f-3007-4c7e-b6e9-5ebc64610484" containerName="revoke-keys" Nov 01 21:01:59.339945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.339920 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d691955f-3007-4c7e-b6e9-5ebc64610484" containerName="revoke-keys" Nov 01 21:01:59.343285 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.343262 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.394889 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.394856 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\"" Nov 01 21:01:59.449817 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.449785 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq"] Nov 01 21:01:59.471834 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.471792 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472027 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.471899 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472027 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.471965 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.472036 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqp8v\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-kube-api-access-wqp8v\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472191 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.472173 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472238 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.472212 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472402 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.472367 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472494 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.472425 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.472544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.472489 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.574170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574131 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.574170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574183 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.574485 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574234 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqp8v\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-kube-api-access-wqp8v\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.574485 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574417 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.574847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574816 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.574946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574823 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.574946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574920 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.575057 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574961 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.575057 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.574997 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.575172 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.575067 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.575172 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.575093 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.575301 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.575241 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.575419 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.575357 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.575615 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.575474 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.577280 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.577256 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.577547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.577531 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.663401 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.661426 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.663401 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.663361 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqp8v\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-kube-api-access-wqp8v\") pod \"maas-default-gateway-openshift-default-58847f9fc5-hqlkq\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:01:59.958233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:01:59.958139 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:02:00.399002 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.398958 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq"] Nov 01 21:02:00.428573 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:02:00.428535 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0da0ce9_0ce9_4609_a2c2_b0a6594b3093.slice/crio-dc05b6264f56c247de7bcd80974abaea92790a3f58ca48af6d7a54b9bc27fbc9 WatchSource:0}: Error finding container dc05b6264f56c247de7bcd80974abaea92790a3f58ca48af6d7a54b9bc27fbc9: Status 404 returned error can't find the container with id dc05b6264f56c247de7bcd80974abaea92790a3f58ca48af6d7a54b9bc27fbc9 Nov 01 21:02:00.431003 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.430962 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:00.431082 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.431064 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:00.431131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.431116 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:00.611690 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.611640 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" event={"ID":"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093","Type":"ContainerStarted","Data":"e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a"} Nov 01 21:02:00.611902 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.611871 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" event={"ID":"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093","Type":"ContainerStarted","Data":"dc05b6264f56c247de7bcd80974abaea92790a3f58ca48af6d7a54b9bc27fbc9"} Nov 01 21:02:00.731241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.731151 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podStartSLOduration=1.731131627 podStartE2EDuration="1.731131627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:00.726610274 +0000 UTC m=+3528.400869529" watchObservedRunningTime="2026-11-01 21:02:00.731131627 +0000 UTC m=+3528.405390884" Nov 01 21:02:00.958453 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:00.958411 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:02:01.960574 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:01.960529 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-hqlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 21:02:01.960985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:01.960583 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 21:02:02.732457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:02.732416 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:02.958901 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:02.958862 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-hqlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 21:02:02.959145 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:02.958927 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 21:02:03.958909 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:03.958875 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-hqlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 21:02:03.959354 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:03.958931 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 21:02:04.958916 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:04.958875 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-hqlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 21:02:04.959436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:04.958937 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 21:02:05.447107 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.447069 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2"] Nov 01 21:02:05.450955 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.450933 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.512900 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.512861 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2"] Nov 01 21:02:05.551170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.551137 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kdm\" (UniqueName: \"kubernetes.io/projected/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-kube-api-access-p2kdm\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.551170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.551173 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.551432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.551258 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.552928 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.552891 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:02:05.552928 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.552919 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:02:05.553141 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.552981 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:02:05.553400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.553379 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:02:05.652792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.652752 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kdm\" (UniqueName: \"kubernetes.io/projected/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-kube-api-access-p2kdm\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.652792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.652791 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.653092 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.652853 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.653614 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.653586 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.655449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.655430 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.712169 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.712132 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kdm\" (UniqueName: \"kubernetes.io/projected/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-kube-api-access-p2kdm\") pod \"maas-api-revoke-sub-a666731f-jwst2\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.764577 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.764539 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:05.958820 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.958770 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-hqlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 21:02:05.959005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:05.958851 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 21:02:06.165478 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:06.165446 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2"] Nov 01 21:02:06.166773 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:02:06.166748 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06d7b40_8fc5_47fb_84f8_8c55c6feeef1.slice/crio-59ec1014c461bcbbfaa80e3ce413a85459cc4530fdf4a9ffa5f0b7a964159943 WatchSource:0}: Error finding container 59ec1014c461bcbbfaa80e3ce413a85459cc4530fdf4a9ffa5f0b7a964159943: Status 404 returned error can't find the container with id 59ec1014c461bcbbfaa80e3ce413a85459cc4530fdf4a9ffa5f0b7a964159943 Nov 01 21:02:06.650042 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:06.649992 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" event={"ID":"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1","Type":"ContainerStarted","Data":"6a67f8c126a956c3e21bfbd43173ed98f7e6ec66a2c132db429e73126772d802"} Nov 01 21:02:06.650250 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:06.650054 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" event={"ID":"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1","Type":"ContainerStarted","Data":"59ec1014c461bcbbfaa80e3ce413a85459cc4530fdf4a9ffa5f0b7a964159943"} Nov 01 21:02:06.764607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:06.764555 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" podStartSLOduration=1.764539734 podStartE2EDuration="1.764539734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:06.761150771 +0000 UTC m=+3534.435410027" watchObservedRunningTime="2026-11-01 21:02:06.764539734 +0000 UTC m=+3534.438798996" Nov 01 21:02:06.960466 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:06.960423 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-hqlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 21:02:06.960869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:06.960484 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 21:02:07.484154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:07.484105 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:07.658139 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:07.658104 5633 generic.go:356] "Generic (PLEG): container finished" podID="a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" containerID="6a67f8c126a956c3e21bfbd43173ed98f7e6ec66a2c132db429e73126772d802" exitCode=0 Nov 01 21:02:07.658364 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:07.658194 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" event={"ID":"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1","Type":"ContainerDied","Data":"6a67f8c126a956c3e21bfbd43173ed98f7e6ec66a2c132db429e73126772d802"} Nov 01 21:02:07.959719 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:07.959636 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-hqlkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 21:02:07.959719 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:07.959704 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 21:02:08.809620 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:08.809576 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:08.989717 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:08.989682 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-xks-ca\") pod \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " Nov 01 21:02:08.989934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:08.989876 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-kube-api-access-p2kdm\" (UniqueName: \"kubernetes.io/projected/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-kube-api-access-p2kdm\") pod \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " Nov 01 21:02:08.990017 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:08.989947 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-openshift-ca\") pod \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\" (UID: \"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1\") " Nov 01 21:02:08.990394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:08.990324 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-openshift-ca" pod "a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" (UID: "a06d7b40-8fc5-47fb-84f8-8c55c6feeef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:08.992137 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:08.992101 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-xks-ca" pod "a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" (UID: "a06d7b40-8fc5-47fb-84f8-8c55c6feeef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:02:08.992280 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:08.992213 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-kube-api-access-p2kdm" pod "a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" (UID: "a06d7b40-8fc5-47fb-84f8-8c55c6feeef1"). InnerVolumeSpecName "kube-api-access-p2kdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:09.091381 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.091310 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2kdm\" (UniqueName: \"kubernetes.io/projected/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-kube-api-access-p2kdm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:09.091381 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.091378 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:09.091381 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.091389 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:09.674379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.674319 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" Nov 01 21:02:09.674379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.674361 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2" event={"ID":"a06d7b40-8fc5-47fb-84f8-8c55c6feeef1","Type":"ContainerDied","Data":"59ec1014c461bcbbfaa80e3ce413a85459cc4530fdf4a9ffa5f0b7a964159943"} Nov 01 21:02:09.674643 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.674414 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ec1014c461bcbbfaa80e3ce413a85459cc4530fdf4a9ffa5f0b7a964159943" Nov 01 21:02:09.927290 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.927203 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:02:09.928015 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.927990 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:02:09.929011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:09.928983 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:02:12.184429 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:12.184392 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:14.447841 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.447803 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9"] Nov 01 21:02:14.448301 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.448288 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" containerName="revoke-keys" Nov 01 21:02:14.448374 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.448302 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" containerName="revoke-keys" Nov 01 21:02:14.448422 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.448395 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" containerName="revoke-keys" Nov 01 21:02:14.451718 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.451701 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544378 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544317 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2nr\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-kube-api-access-hm2nr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544580 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544428 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544580 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544481 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544580 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544518 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544648 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544677 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544768 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544698 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544808 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544781 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bddec4-65c4-4adf-ae35-3894be629d16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.544865 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.544852 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.551964 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.551935 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9"] Nov 01 21:02:14.645629 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.645587 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.645851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.645642 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2nr\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-kube-api-access-hm2nr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.645851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.645671 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.645851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.645719 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.645851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.645762 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.645851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.645813 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.645851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.645845 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.646171 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.646027 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.646171 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.646116 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bddec4-65c4-4adf-ae35-3894be629d16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.646291 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.646200 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.646291 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.646234 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.646469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.646286 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.646521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.646468 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.646757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.646740 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bddec4-65c4-4adf-ae35-3894be629d16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.648260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.648236 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.648421 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.648308 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.702205 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.702132 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2nr\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-kube-api-access-hm2nr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.702205 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.702154 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f66n9\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:14.765491 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:14.765447 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:15.190122 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.190091 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9"] Nov 01 21:02:15.191423 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:02:15.191390 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bddec4_65c4_4adf_ae35_3894be629d16.slice/crio-44dfbf38c2a6e6be733dba04a5c00105ad7a105a1cae6046fa3df90168adf545 WatchSource:0}: Error finding container 44dfbf38c2a6e6be733dba04a5c00105ad7a105a1cae6046fa3df90168adf545: Status 404 returned error can't find the container with id 44dfbf38c2a6e6be733dba04a5c00105ad7a105a1cae6046fa3df90168adf545 Nov 01 21:02:15.193901 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.193865 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:15.194026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.193952 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:15.194026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.193995 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:15.717037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.716991 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" event={"ID":"c2bddec4-65c4-4adf-ae35-3894be629d16","Type":"ContainerStarted","Data":"655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b"} Nov 01 21:02:15.717516 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.717067 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" event={"ID":"c2bddec4-65c4-4adf-ae35-3894be629d16","Type":"ContainerStarted","Data":"44dfbf38c2a6e6be733dba04a5c00105ad7a105a1cae6046fa3df90168adf545"} Nov 01 21:02:15.765686 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.765652 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:15.836706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.836641 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podStartSLOduration=1.836625768 podStartE2EDuration="1.836625768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:15.833968145 +0000 UTC m=+3543.508227400" watchObservedRunningTime="2026-11-01 21:02:15.836625768 +0000 UTC m=+3543.510885022" Nov 01 21:02:15.912994 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:15.912958 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:16.156717 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:02:16.156625 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:02:16.767367 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:16.767312 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:16.767772 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:16.767378 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:16.918376 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:16.918325 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:17.535134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:17.535097 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:17.662198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:17.662157 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:17.766697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:17.766658 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:17.766895 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:17.766731 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:18.765923 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:18.765870 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:18.766310 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:18.765953 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:19.766811 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:19.766778 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:19.767295 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:19.767263 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:19.883442 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:19.883398 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:20.112761 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:20.106953 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:20.424184 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:20.424092 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:02:20.766661 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:20.766626 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:20.766854 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:20.766679 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:21.766060 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:21.766021 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:21.766060 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:21.766077 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:21.792700 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:21.792659 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:22.769355 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:22.769199 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:22.769355 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:22.769263 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:23.765893 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:23.765859 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f66n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Nov 01 21:02:23.766301 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:23.765922 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Nov 01 21:02:23.809089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:23.808854 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:25.389794 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:25.389758 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:25.390462 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:25.390395 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:25.391691 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:25.391668 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:02:25.980271 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:25.980223 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:27.323528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:27.323480 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:27.940355 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:27.939787 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:31.278055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:31.278017 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:32.173830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:32.173788 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:32.294041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:32.294004 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:33.899794 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:33.899759 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:36.002251 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:36.002207 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:37.749986 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:37.749949 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:38.658720 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:38.658678 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:38.838646 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:38.838499 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:38.958613 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:38.958501 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:40.695944 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:40.695904 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:40.867910 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:40.867867 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:44.176221 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.176126 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:02:44.702213 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.702170 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn"] Nov 01 21:02:44.706089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.706062 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.809392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.809353 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn"] Nov 01 21:02:44.827096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.827057 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:44.833925 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.833882 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.833984 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.834022 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmth\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-kube-api-access-qqmth\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.834092 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.834161 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.834215 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.834285 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.834353 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.834648 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.834400 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935326 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935288 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935326 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935342 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmth\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-kube-api-access-qqmth\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935400 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935443 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935485 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935540 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935580 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935623 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935958 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935646 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.935958 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.935865 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.936433 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.936400 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.936548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.936436 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.936548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.936413 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.936626 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.936601 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.938000 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.937973 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.939645 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.939616 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.992144 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.992104 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:44.992316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:44.992148 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmth\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-kube-api-access-qqmth\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4rtsn\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:45.022044 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.022006 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:45.464472 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.464442 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn"] Nov 01 21:02:45.467865 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:02:45.467838 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f0b64c_9675_47af_b0d7_185ea0b8afa1.slice/crio-88401427b441a361b459be6e46df9f22870fdd0c775092223e1f339f65474ead WatchSource:0}: Error finding container 88401427b441a361b459be6e46df9f22870fdd0c775092223e1f339f65474ead: Status 404 returned error can't find the container with id 88401427b441a361b459be6e46df9f22870fdd0c775092223e1f339f65474ead Nov 01 21:02:45.470985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.470945 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:45.471082 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.471046 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:45.471123 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.471090 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:02:45.496896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.496861 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:45.972863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.972823 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" event={"ID":"72f0b64c-9675-47af-b0d7-185ea0b8afa1","Type":"ContainerStarted","Data":"371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720"} Nov 01 21:02:45.973110 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:45.972917 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" event={"ID":"72f0b64c-9675-47af-b0d7-185ea0b8afa1","Type":"ContainerStarted","Data":"88401427b441a361b459be6e46df9f22870fdd0c775092223e1f339f65474ead"} Nov 01 21:02:46.041358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:46.035170 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:46.041358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:46.035420 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:46.041358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:46.035509 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:46.104955 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:46.104873 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podStartSLOduration=2.104849906 podStartE2EDuration="2.104849906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:46.101037575 +0000 UTC m=+3573.775296824" watchObservedRunningTime="2026-11-01 21:02:46.104849906 +0000 UTC m=+3573.779109162" Nov 01 21:02:46.235898 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:02:46.235765 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:02:47.036532 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:47.036492 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:47.037201 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:47.037158 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:47.234993 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:47.234953 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:47.355800 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:47.355700 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:48.042361 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:48.035635 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:48.042361 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:48.035698 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:48.953477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:48.953437 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:49.022424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:49.022389 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:49.022621 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:49.022445 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:49.913114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:49.913075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:50.023701 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:50.023662 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:50.023929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:50.023728 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:50.033158 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:50.033128 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:50.176809 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:50.176713 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:50.419148 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:50.419111 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:50.542318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:50.542279 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:51.023306 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:51.023274 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:51.023765 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:51.023357 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:51.203358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:51.202010 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:51.329607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:51.329521 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:52.022970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:52.022933 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:52.023314 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:52.023263 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:53.022525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:53.022482 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 01 21:02:53.022726 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:53.022552 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 01 21:02:54.977909 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:54.977872 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:55.064523 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:55.064480 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4rtsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:02:55.064740 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:55.064537 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:02:55.205681 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:55.205648 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:55.206112 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:55.206068 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:55.207210 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:55.207186 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:02:55.293254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:55.293154 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:56.899064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:56.899019 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:02:59.812294 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.812263 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d"] Nov 01 21:02:59.817245 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.817224 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850406 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850458 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850552 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850577 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9f9\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-kube-api-access-bp9f9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850610 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850775 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850707 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850775 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850751 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850862 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850844 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.850909 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.850898 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.882694 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.882657 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:02:59.912627 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.912595 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d"] Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953563 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953622 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953669 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953693 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9f9\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-kube-api-access-bp9f9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953739 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953768 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953814 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953875 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.953930 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.954373 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.954952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.954816 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.955719 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.955483 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.955864 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.955836 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.956474 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.956449 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.958363 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.958298 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:02:59.962494 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:02:59.962467 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:00.011065 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:00.011030 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9f9\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-kube-api-access-bp9f9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:00.011230 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:00.011033 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8ld9d\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:00.132515 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:00.132423 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:00.562737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:00.562698 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d"] Nov 01 21:03:00.563524 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:03:00.563494 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a167059_09af_4a4b_a89b_dd8f0f3cde14.slice/crio-6d202f9e4bb1fef806258cdbfddf09083d63d5685b7a65ce74dac2871b50b423 WatchSource:0}: Error finding container 6d202f9e4bb1fef806258cdbfddf09083d63d5685b7a65ce74dac2871b50b423: Status 404 returned error can't find the container with id 6d202f9e4bb1fef806258cdbfddf09083d63d5685b7a65ce74dac2871b50b423 Nov 01 21:03:00.566022 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:00.565980 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:03:00.566137 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:00.566079 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:03:00.566137 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:00.566126 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:03:01.126501 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:01.126462 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" event={"ID":"4a167059-09af-4a4b-a89b-dd8f0f3cde14","Type":"ContainerStarted","Data":"c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd"} Nov 01 21:03:01.127047 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:01.126558 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" event={"ID":"4a167059-09af-4a4b-a89b-dd8f0f3cde14","Type":"ContainerStarted","Data":"6d202f9e4bb1fef806258cdbfddf09083d63d5685b7a65ce74dac2871b50b423"} Nov 01 21:03:01.133044 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:01.133007 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:01.134839 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:01.134817 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:01.134897 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:01.134867 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:01.332814 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:01.332755 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podStartSLOduration=2.332739648 podStartE2EDuration="2.332739648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:01.290854519 +0000 UTC m=+3588.965113774" watchObservedRunningTime="2026-11-01 21:03:01.332739648 +0000 UTC m=+3589.006998902" Nov 01 21:03:02.132850 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:02.132810 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:02.133235 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:02.132865 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:03.132974 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:03.132940 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:03.133453 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:03.133009 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:04.133121 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:04.133089 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:04.133594 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:04.133142 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:04.611134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:04.611084 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:05.133171 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:05.133127 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:05.133627 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:05.133193 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:06.133087 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:06.133045 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:06.133319 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:06.133108 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:07.133166 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:07.133129 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:07.133353 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:07.133197 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:08.133760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:08.133726 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8ld9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 01 21:03:08.134170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:08.133777 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 01 21:03:09.380070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:09.380024 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:09.891484 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:09.891450 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:09.892042 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:09.892018 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:09.892882 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:09.892865 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:03:14.086157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:14.086107 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:15.768395 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:15.768351 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:03:16.156865 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:03:16.156783 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:03:18.789920 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:18.789860 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:19.081250 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:19.081156 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:19.204476 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:19.204429 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:20.471415 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:20.471381 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:23.084123 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:23.084038 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:23.582669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:23.582632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:28.383814 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:28.383774 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:30.066756 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:30.066719 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:03:33.132864 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:33.132821 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:37.811149 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:37.811112 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:38.174654 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:38.174559 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:40.379502 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:40.379470 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:03:42.477729 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:42.477686 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:44.787599 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.787561 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v"] Nov 01 21:03:44.791391 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.791371 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:44.847712 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.847676 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v"] Nov 01 21:03:44.892249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.892219 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:03:44.892474 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.892440 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:03:44.892474 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.892460 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:03:44.893018 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.893004 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:03:44.967369 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.967303 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:44.967570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.967447 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsr5t\" (UniqueName: \"kubernetes.io/projected/3702827b-4f17-4269-9d92-45d06aba0846-kube-api-access-zsr5t\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:44.967570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:44.967503 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.068363 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.068257 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.068579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.068389 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.068579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.068443 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsr5t\" (UniqueName: \"kubernetes.io/projected/3702827b-4f17-4269-9d92-45d06aba0846-kube-api-access-zsr5t\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.069161 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.069139 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.071008 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.070975 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.126547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.126504 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsr5t\" (UniqueName: \"kubernetes.io/projected/3702827b-4f17-4269-9d92-45d06aba0846-kube-api-access-zsr5t\") pod \"maas-api-revoke-sub-49bf212a-q788v\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.404855 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.404777 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:45.778241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:45.778209 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v"] Nov 01 21:03:45.779069 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:03:45.779029 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3702827b_4f17_4269_9d92_45d06aba0846.slice/crio-721f94e3bc7d195b7927ccd86000560a8763aba1fd344a521a0bc5b6594f2743 WatchSource:0}: Error finding container 721f94e3bc7d195b7927ccd86000560a8763aba1fd344a521a0bc5b6594f2743: Status 404 returned error can't find the container with id 721f94e3bc7d195b7927ccd86000560a8763aba1fd344a521a0bc5b6594f2743 Nov 01 21:03:46.157398 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:03:46.157351 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:03:46.409557 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:46.409460 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" event={"ID":"3702827b-4f17-4269-9d92-45d06aba0846","Type":"ContainerStarted","Data":"22311c95b9e495965f79a9e26a5f54193fce5a1980e89d8982979bf534b68ee3"} Nov 01 21:03:46.409557 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:46.409541 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" event={"ID":"3702827b-4f17-4269-9d92-45d06aba0846","Type":"ContainerStarted","Data":"721f94e3bc7d195b7927ccd86000560a8763aba1fd344a521a0bc5b6594f2743"} Nov 01 21:03:46.528462 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:46.528375 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" podStartSLOduration=2.528351578 podStartE2EDuration="2.528351578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:46.523005516 +0000 UTC m=+3634.197264769" watchObservedRunningTime="2026-11-01 21:03:46.528351578 +0000 UTC m=+3634.202610834" Nov 01 21:03:47.186083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:47.186049 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:47.417382 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:47.417326 5633 generic.go:356] "Generic (PLEG): container finished" podID="3702827b-4f17-4269-9d92-45d06aba0846" containerID="22311c95b9e495965f79a9e26a5f54193fce5a1980e89d8982979bf534b68ee3" exitCode=0 Nov 01 21:03:47.417588 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:47.417425 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" event={"ID":"3702827b-4f17-4269-9d92-45d06aba0846","Type":"ContainerDied","Data":"22311c95b9e495965f79a9e26a5f54193fce5a1980e89d8982979bf534b68ee3"} Nov 01 21:03:48.565945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.565914 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:48.708607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.708506 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-xks-ca\") pod \"3702827b-4f17-4269-9d92-45d06aba0846\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " Nov 01 21:03:48.708607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.708556 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-openshift-ca\") pod \"3702827b-4f17-4269-9d92-45d06aba0846\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " Nov 01 21:03:48.708867 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.708717 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3702827b-4f17-4269-9d92-45d06aba0846-kube-api-access-zsr5t\" (UniqueName: \"kubernetes.io/projected/3702827b-4f17-4269-9d92-45d06aba0846-kube-api-access-zsr5t\") pod \"3702827b-4f17-4269-9d92-45d06aba0846\" (UID: \"3702827b-4f17-4269-9d92-45d06aba0846\") " Nov 01 21:03:48.709048 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.709020 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-openshift-ca" pod "3702827b-4f17-4269-9d92-45d06aba0846" (UID: "3702827b-4f17-4269-9d92-45d06aba0846"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:03:48.709187 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.709168 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:48.711107 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.711082 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-xks-ca" pod "3702827b-4f17-4269-9d92-45d06aba0846" (UID: "3702827b-4f17-4269-9d92-45d06aba0846"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:03:48.711256 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.711152 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3702827b-4f17-4269-9d92-45d06aba0846-kube-api-access-zsr5t" pod "3702827b-4f17-4269-9d92-45d06aba0846" (UID: "3702827b-4f17-4269-9d92-45d06aba0846"). InnerVolumeSpecName "kube-api-access-zsr5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:03:48.809921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.809884 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3702827b-4f17-4269-9d92-45d06aba0846-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:48.809921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:48.809917 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsr5t\" (UniqueName: \"kubernetes.io/projected/3702827b-4f17-4269-9d92-45d06aba0846-kube-api-access-zsr5t\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:03:49.433499 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:49.433459 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" event={"ID":"3702827b-4f17-4269-9d92-45d06aba0846","Type":"ContainerDied","Data":"721f94e3bc7d195b7927ccd86000560a8763aba1fd344a521a0bc5b6594f2743"} Nov 01 21:03:49.433682 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:49.433524 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721f94e3bc7d195b7927ccd86000560a8763aba1fd344a521a0bc5b6594f2743" Nov 01 21:03:49.433682 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:49.433471 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v" Nov 01 21:03:51.887257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:51.887227 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:56.594693 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:56.594651 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:56.955543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:56.955449 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:57.084170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.084134 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:57.223623 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.223580 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:03:57.775451 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.775414 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6"] Nov 01 21:03:57.776612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.776581 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3702827b-4f17-4269-9d92-45d06aba0846" containerName="revoke-keys" Nov 01 21:03:57.776612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.776611 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3702827b-4f17-4269-9d92-45d06aba0846" containerName="revoke-keys" Nov 01 21:03:57.776841 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.776723 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3702827b-4f17-4269-9d92-45d06aba0846" containerName="revoke-keys" Nov 01 21:03:57.780385 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.780364 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.799005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.798970 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcl8\" (UniqueName: \"kubernetes.io/projected/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-kube-api-access-glcl8\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.799200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.799181 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.799247 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.799218 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.832469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.832433 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6"] Nov 01 21:03:57.876689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.876658 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:03:57.881281 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.881249 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:03:57.881281 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.881246 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:03:57.881281 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.881252 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:03:57.900226 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.900193 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.900475 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.900249 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.900475 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.900294 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcl8\" (UniqueName: \"kubernetes.io/projected/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-kube-api-access-glcl8\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.901035 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.901011 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.902743 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.902723 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:57.956696 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:57.956669 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcl8\" (UniqueName: \"kubernetes.io/projected/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-kube-api-access-glcl8\") pod \"maas-api-revoke-sub-9471878a-8cpn6\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:58.094511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:58.094413 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:03:58.468222 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:58.468191 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6"] Nov 01 21:03:58.468871 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:03:58.468846 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac3edcf_e108_4b3d_b3ae_9d93a7160585.slice/crio-0429aca776292d5e5ad49184d46a87354c5097dc81afc00a6ce226b9766d813b WatchSource:0}: Error finding container 0429aca776292d5e5ad49184d46a87354c5097dc81afc00a6ce226b9766d813b: Status 404 returned error can't find the container with id 0429aca776292d5e5ad49184d46a87354c5097dc81afc00a6ce226b9766d813b Nov 01 21:03:58.509412 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:58.509371 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" event={"ID":"3ac3edcf-e108-4b3d-b3ae-9d93a7160585","Type":"ContainerStarted","Data":"0429aca776292d5e5ad49184d46a87354c5097dc81afc00a6ce226b9766d813b"} Nov 01 21:03:59.517000 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:59.516959 5633 generic.go:356] "Generic (PLEG): container finished" podID="3ac3edcf-e108-4b3d-b3ae-9d93a7160585" containerID="b062c05932d3b548a997e650b91c3fafd5aaf058a0d532e97c75ee080ee48e63" exitCode=0 Nov 01 21:03:59.517465 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:59.517061 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" event={"ID":"3ac3edcf-e108-4b3d-b3ae-9d93a7160585","Type":"ContainerDied","Data":"b062c05932d3b548a997e650b91c3fafd5aaf058a0d532e97c75ee080ee48e63"} Nov 01 21:03:59.795432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:03:59.795394 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:04:00.673146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.673118 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:04:00.731792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.731753 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-kube-api-access-glcl8\" (UniqueName: \"kubernetes.io/projected/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-kube-api-access-glcl8\") pod \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " Nov 01 21:04:00.732011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.731843 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-openshift-ca\") pod \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " Nov 01 21:04:00.732011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.731884 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-xks-ca\") pod \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\" (UID: \"3ac3edcf-e108-4b3d-b3ae-9d93a7160585\") " Nov 01 21:04:00.732260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.732235 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-openshift-ca" pod "3ac3edcf-e108-4b3d-b3ae-9d93a7160585" (UID: "3ac3edcf-e108-4b3d-b3ae-9d93a7160585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:00.734178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.734155 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-xks-ca" pod "3ac3edcf-e108-4b3d-b3ae-9d93a7160585" (UID: "3ac3edcf-e108-4b3d-b3ae-9d93a7160585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:04:00.734257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.734175 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-kube-api-access-glcl8" pod "3ac3edcf-e108-4b3d-b3ae-9d93a7160585" (UID: "3ac3edcf-e108-4b3d-b3ae-9d93a7160585"). InnerVolumeSpecName "kube-api-access-glcl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:00.833667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.833573 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glcl8\" (UniqueName: \"kubernetes.io/projected/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-kube-api-access-glcl8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:00.833667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.833606 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:00.833667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:00.833620 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac3edcf-e108-4b3d-b3ae-9d93a7160585-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:01.282730 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:01.282696 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:01.531146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:01.531114 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" Nov 01 21:04:01.531420 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:01.531123 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6" event={"ID":"3ac3edcf-e108-4b3d-b3ae-9d93a7160585","Type":"ContainerDied","Data":"0429aca776292d5e5ad49184d46a87354c5097dc81afc00a6ce226b9766d813b"} Nov 01 21:04:01.531420 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:01.531230 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0429aca776292d5e5ad49184d46a87354c5097dc81afc00a6ce226b9766d813b" Nov 01 21:04:06.068240 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:06.068196 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:09.920678 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:09.920636 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:10.141670 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:10.141632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:10.266726 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:10.266685 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:10.684308 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:10.684195 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:10.803832 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:10.803790 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:13.501769 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:13.501730 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:04:15.483404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:15.483366 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:16.158006 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:04:16.157963 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:04:16.384908 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:16.384871 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:16.523489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:16.523456 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:16.689156 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:16.689103 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:16.837642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:16.837562 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:17.075918 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:17.075870 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:17.896344 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:17.896300 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:18.065795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:18.065763 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:20.369676 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:20.369634 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:21.866033 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:21.865999 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:24.386816 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:24.386778 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:25.082765 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:25.082732 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:29.073063 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:29.073018 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:04:29.790319 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:29.790268 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:34.507787 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:34.507752 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:38.434502 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:38.434463 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:39.175954 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:39.175912 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:43.877777 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:43.877735 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:46.156869 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:04:46.156832 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:04:48.599993 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:48.599951 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:53.285448 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:53.285324 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:54.418163 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:54.418126 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:04:56.982009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:56.981974 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:04:57.997291 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:04:57.997246 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:02.767115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:02.767073 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:07.467407 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:07.467377 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:12.044206 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:12.044167 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj"] Nov 01 21:05:12.093488 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:12.093451 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d44c61-bcrvj"] Nov 01 21:05:12.167526 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:12.167487 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:12.466165 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:12.466131 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76a9d29-6cb7-4d7c-b903-9f56ae84b91d" path="/var/lib/kubelet/pods/e76a9d29-6cb7-4d7c-b903-9f56ae84b91d/volumes" Nov 01 21:05:16.159305 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:05:16.159265 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:05:16.885350 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:16.885302 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:21.575846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:21.575806 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:25.081851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:25.081816 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6"] Nov 01 21:05:25.130045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:25.130005 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f116018-ncvg6"] Nov 01 21:05:26.296930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:26.296896 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:26.466344 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:26.466306 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc119687-432b-464e-8694-67981bd29159" path="/var/lib/kubelet/pods/cc119687-432b-464e-8694-67981bd29159/volumes" Nov 01 21:05:26.871739 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:26.871696 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:05:30.989931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:30.989898 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:31.164751 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:31.164710 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:31.364744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:31.364664 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:35.689306 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:35.689269 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:37.045587 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:37.045551 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w"] Nov 01 21:05:37.048498 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:37.048474 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d395028-2gc2w"] Nov 01 21:05:38.466983 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:38.466950 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d3f245-50b9-4509-bd6e-a313a8b563ca" path="/var/lib/kubelet/pods/f0d3f245-50b9-4509-bd6e-a313a8b563ca/volumes" Nov 01 21:05:38.811570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:38.811474 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:40.381736 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:40.381697 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:41.117081 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:41.117043 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:05:45.098642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:45.098607 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:46.157151 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:05:46.157110 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:05:49.808231 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:49.808192 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:52.045721 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:52.045681 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg"] Nov 01 21:05:52.093140 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:52.093100 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe2dfdd7-wxzvg"] Nov 01 21:05:52.466215 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:52.466179 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347fe460-132d-4582-a1bd-4ab20aa03e47" path="/var/lib/kubelet/pods/347fe460-132d-4582-a1bd-4ab20aa03e47/volumes" Nov 01 21:05:53.673487 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:53.673446 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:05:54.640803 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:54.640760 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:05:59.414475 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:05:59.414437 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:02.921932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:02.921897 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:04.105419 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:04.105366 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:04.598052 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:04.598014 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:06:06.045421 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:06.045388 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l"] Nov 01 21:06:06.094599 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:06.094560 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac7d3ee-twr9l"] Nov 01 21:06:06.466283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:06.466241 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d691955f-3007-4c7e-b6e9-5ebc64610484" path="/var/lib/kubelet/pods/d691955f-3007-4c7e-b6e9-5ebc64610484/volumes" Nov 01 21:06:08.888862 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:08.888808 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:13.601321 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:13.601261 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:16.157832 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:06:16.157788 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:06:18.148943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:18.148907 5633 scope.go:122] "RemoveContainer" containerID="7620f068bd8eadaf412c092ea0dec98696682ab22aab4b4d7b2b9b0bf0816c8a" Nov 01 21:06:18.284948 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:18.284898 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:23.042238 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:23.042077 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:25.615228 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:25.615184 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:25.745394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:25.745320 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:27.698006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:27.697963 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:32.459116 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:32.459077 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:37.084449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:37.084395 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:39.116610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:39.116572 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:41.792687 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:41.792632 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:46.157052 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:06:46.157018 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:06:46.480457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:46.480414 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:51.181575 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:51.181533 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:06:55.894856 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:06:55.894811 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:00.586784 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:00.586743 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:05.311548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:05.311510 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:09.049321 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:09.049284 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2"] Nov 01 21:07:09.098611 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:09.098569 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a666731f-jwst2"] Nov 01 21:07:09.985994 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:09.985942 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:10.466425 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:10.466393 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06d7b40-8fc5-47fb-84f8-8c55c6feeef1" path="/var/lib/kubelet/pods/a06d7b40-8fc5-47fb-84f8-8c55c6feeef1/volumes" Nov 01 21:07:14.692380 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:14.692310 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:16.157009 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:07:16.156968 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:07:18.177210 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:18.177180 5633 scope.go:122] "RemoveContainer" containerID="b96f6543e4fd068a6e0db92fe01ef1e4bcaec380f1a7f0d92ceec3c63d8136fc" Nov 01 21:07:18.186694 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:18.186672 5633 scope.go:122] "RemoveContainer" containerID="b7eb754c868eea17247a326bf2898c587748c258fdbcde91e25ece676e6c2ba4" Nov 01 21:07:18.195943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:18.195919 5633 scope.go:122] "RemoveContainer" containerID="9a9da119ec88375fd33c7c8c7a16583cdb266070f39485fac70db1e2e710799c" Nov 01 21:07:18.204759 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:18.204732 5633 scope.go:122] "RemoveContainer" containerID="37a8861eeddc8a736995f30aed5c4bcff19cbecac228578b5f5acead17573ec7" Nov 01 21:07:19.400234 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:19.400193 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:24.138666 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:24.138629 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:28.777879 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:28.777840 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:33.451417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:33.451378 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:38.182750 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:38.182702 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:39.481742 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:39.481698 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:42.885623 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:42.885588 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:46.158498 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:07:46.158455 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:07:47.599290 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:47.599252 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:52.397363 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:52.397251 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:55.140772 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:55.140727 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:07:57.063728 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:07:57.063691 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:01.722757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:01.722717 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:06.430495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:06.430454 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:11.078504 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:11.078467 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:11.262169 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:11.262130 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:15.775771 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:15.775715 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:16.156961 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:08:16.156859 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:08:18.241617 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:18.241586 5633 scope.go:122] "RemoveContainer" containerID="6a67f8c126a956c3e21bfbd43173ed98f7e6ec66a2c132db429e73126772d802" Nov 01 21:08:20.476685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:20.476654 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:25.173186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:25.173154 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:29.879864 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:29.879821 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:32.128635 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.128596 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9"] Nov 01 21:08:32.129185 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.128876 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" containerID="cri-o://655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b" gracePeriod=30 Nov 01 21:08:32.178906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.178863 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq"] Nov 01 21:08:32.179152 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.179126 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" containerID="cri-o://e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a" gracePeriod=30 Nov 01 21:08:32.181814 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.181783 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d"] Nov 01 21:08:32.182045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.181995 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" containerID="cri-o://c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd" gracePeriod=30 Nov 01 21:08:32.228376 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.228343 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn"] Nov 01 21:08:32.228586 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:32.228565 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" containerID="cri-o://371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720" gracePeriod=30 Nov 01 21:08:34.630404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:34.630319 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:37.460855 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.460828 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:08:37.507740 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.507711 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:08:37.557737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557697 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-credential-socket\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.557737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557744 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-credential-socket\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.557996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557785 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-data\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.557996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557809 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istiod-ca-cert\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.557996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557834 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-token\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-token\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.557996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557877 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-podinfo\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.557996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557951 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-certs\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.557996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557980 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-kube-api-access-hm2nr\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-kube-api-access-hm2nr\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.557999 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-credential-socket" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558041 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-envoy\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558105 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-envoy\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558095 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-data" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558137 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-podinfo\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558154 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-credential-socket" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558204 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-certs\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558238 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-kube-api-access-bp9f9\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-kube-api-access-bp9f9\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558247 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istiod-ca-cert" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558263 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-certs" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558271 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bddec4-65c4-4adf-ae35-3894be629d16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bddec4-65c4-4adf-ae35-3894be629d16-istiod-ca-cert\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558311 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-socket\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.558377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558362 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-socket\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558415 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-data\") pod \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\" (UID: \"4a167059-09af-4a4b-a89b-dd8f0f3cde14\") " Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558451 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-token\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-token\") pod \"c2bddec4-65c4-4adf-ae35-3894be629d16\" (UID: \"c2bddec4-65c4-4adf-ae35-3894be629d16\") " Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558514 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-certs" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558682 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-socket" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558945 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558965 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558981 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.558996 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.559009 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.559027 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.559043 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.559782 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-socket" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.560384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.559981 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bddec4-65c4-4adf-ae35-3894be629d16-istiod-ca-cert" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:37.562156 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.562128 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-data" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.564651 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.564623 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-podinfo" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:08:37.565257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.565211 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-envoy" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.565374 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.565267 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-podinfo" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:08:37.565458 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.565427 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-envoy" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.566074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.566043 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-kube-api-access-hm2nr" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "kube-api-access-hm2nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.567275 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.567243 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-token" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.567662 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.567643 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-token" pod "c2bddec4-65c4-4adf-ae35-3894be629d16" (UID: "c2bddec4-65c4-4adf-ae35-3894be629d16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.568585 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.568557 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-kube-api-access-bp9f9" pod "4a167059-09af-4a4b-a89b-dd8f0f3cde14" (UID: "4a167059-09af-4a4b-a89b-dd8f0f3cde14"). InnerVolumeSpecName "kube-api-access-bp9f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.580795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.580769 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:08:37.590905 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.590876 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:08:37.659427 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659389 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-data\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.659630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659454 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-token\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-token\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.659630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659513 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-podinfo\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.659630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659564 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istiod-ca-cert\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.659760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659629 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-credential-socket\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.659760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659678 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-socket\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.659760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659703 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-data" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.659760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659725 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-certs\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.659983 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659764 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-certs\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.659983 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659801 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-token\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-token\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.659983 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659832 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-envoy\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.659983 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659870 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-envoy\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.659983 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659892 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-credential-socket" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.659983 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.659903 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istiod-ca-cert\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.660320 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660020 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-socket" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.660320 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660038 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-certs" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.660320 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660235 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istiod-ca-cert" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:37.660320 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660294 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-socket\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660361 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-kube-api-access-wqp8v\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-kube-api-access-wqp8v\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660395 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-podinfo\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660432 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-data\") pod \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\" (UID: \"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093\") " Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660461 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-credential-socket\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660491 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-kube-api-access-qqmth\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-kube-api-access-qqmth\") pod \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\" (UID: \"72f0b64c-9675-47af-b0d7-185ea0b8afa1\") " Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660597 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-socket" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660701 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-certs" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660795 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-data" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660869 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-credential-socket" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.660967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.660898 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istiod-ca-cert" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661180 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661203 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661218 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661233 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661247 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661262 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661275 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661288 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661302 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661317 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm2nr\" (UniqueName: \"kubernetes.io/projected/c2bddec4-65c4-4adf-ae35-3894be629d16-kube-api-access-hm2nr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661351 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661364 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661379 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661391 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661406 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661422 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661463 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bddec4-65c4-4adf-ae35-3894be629d16-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661478 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661494 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9f9\" (UniqueName: \"kubernetes.io/projected/4a167059-09af-4a4b-a89b-dd8f0f3cde14-kube-api-access-bp9f9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661509 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bddec4-65c4-4adf-ae35-3894be629d16-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.661745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.661525 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a167059-09af-4a4b-a89b-dd8f0f3cde14-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.662967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.662611 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-token" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.662967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.662620 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-podinfo" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:08:37.663525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.663499 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-kube-api-access-wqp8v" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "kube-api-access-wqp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.663628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.663524 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-kube-api-access-qqmth" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "kube-api-access-qqmth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.663628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.663546 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-token" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:08:37.663628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.663604 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-envoy" pod "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" (UID: "a0da0ce9-0ce9-4609-a2c2-b0a6594b3093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.663797 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.663781 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-envoy" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:08:37.664866 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.664846 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-podinfo" pod "72f0b64c-9675-47af-b0d7-185ea0b8afa1" (UID: "72f0b64c-9675-47af-b0d7-185ea0b8afa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:08:37.762435 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762389 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.762435 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762419 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.762435 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762431 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.762435 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762440 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.762435 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762449 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.762878 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762457 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqp8v\" (UniqueName: \"kubernetes.io/projected/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093-kube-api-access-wqp8v\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.762878 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762466 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0b64c-9675-47af-b0d7-185ea0b8afa1-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:37.762878 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:37.762475 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmth\" (UniqueName: \"kubernetes.io/projected/72f0b64c-9675-47af-b0d7-185ea0b8afa1-kube-api-access-qqmth\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:08:38.212091 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.212049 5633 generic.go:356] "Generic (PLEG): container finished" podID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerID="371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720" exitCode=0 Nov 01 21:08:38.212293 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.212107 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" Nov 01 21:08:38.212293 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.212131 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" event={"ID":"72f0b64c-9675-47af-b0d7-185ea0b8afa1","Type":"ContainerDied","Data":"371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720"} Nov 01 21:08:38.212293 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.212189 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn" event={"ID":"72f0b64c-9675-47af-b0d7-185ea0b8afa1","Type":"ContainerDied","Data":"88401427b441a361b459be6e46df9f22870fdd0c775092223e1f339f65474ead"} Nov 01 21:08:38.212293 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.212203 5633 scope.go:122] "RemoveContainer" containerID="371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720" Nov 01 21:08:38.213722 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.213696 5633 generic.go:356] "Generic (PLEG): container finished" podID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerID="655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b" exitCode=0 Nov 01 21:08:38.213825 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.213750 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" Nov 01 21:08:38.213825 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.213765 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" event={"ID":"c2bddec4-65c4-4adf-ae35-3894be629d16","Type":"ContainerDied","Data":"655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b"} Nov 01 21:08:38.213948 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.213832 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9" event={"ID":"c2bddec4-65c4-4adf-ae35-3894be629d16","Type":"ContainerDied","Data":"44dfbf38c2a6e6be733dba04a5c00105ad7a105a1cae6046fa3df90168adf545"} Nov 01 21:08:38.215395 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.215370 5633 generic.go:356] "Generic (PLEG): container finished" podID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerID="c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd" exitCode=0 Nov 01 21:08:38.215524 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.215429 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" Nov 01 21:08:38.215524 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.215461 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" event={"ID":"4a167059-09af-4a4b-a89b-dd8f0f3cde14","Type":"ContainerDied","Data":"c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd"} Nov 01 21:08:38.215524 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.215504 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d" event={"ID":"4a167059-09af-4a4b-a89b-dd8f0f3cde14","Type":"ContainerDied","Data":"6d202f9e4bb1fef806258cdbfddf09083d63d5685b7a65ce74dac2871b50b423"} Nov 01 21:08:38.217045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.217025 5633 generic.go:356] "Generic (PLEG): container finished" podID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerID="e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a" exitCode=0 Nov 01 21:08:38.217155 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.217076 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" Nov 01 21:08:38.217155 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.217067 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" event={"ID":"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093","Type":"ContainerDied","Data":"e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a"} Nov 01 21:08:38.217232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.217195 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq" event={"ID":"a0da0ce9-0ce9-4609-a2c2-b0a6594b3093","Type":"ContainerDied","Data":"dc05b6264f56c247de7bcd80974abaea92790a3f58ca48af6d7a54b9bc27fbc9"} Nov 01 21:08:38.225444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.225413 5633 scope.go:122] "RemoveContainer" containerID="371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720" Nov 01 21:08:38.225765 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:08:38.225729 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720\": container with ID starting with 371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720 not found: ID does not exist" containerID="371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720" Nov 01 21:08:38.225960 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.225766 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720"} err="failed to get container status \"371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720\": rpc error: code = NotFound desc = could not find container \"371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720\": container with ID starting with 371de7d610f1e51541641377fc56e2b670fd9788cdfacc25a9faec9f0c056720 not found: ID does not exist" Nov 01 21:08:38.225960 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.225783 5633 scope.go:122] "RemoveContainer" containerID="655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b" Nov 01 21:08:38.237109 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.237086 5633 scope.go:122] "RemoveContainer" containerID="655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b" Nov 01 21:08:38.237442 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:08:38.237418 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b\": container with ID starting with 655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b not found: ID does not exist" containerID="655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b" Nov 01 21:08:38.237519 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.237450 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b"} err="failed to get container status \"655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b\": rpc error: code = NotFound desc = could not find container \"655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b\": container with ID starting with 655aab39c7685e6426547a2fd77a46a60ea6be150a6228931a9894b38fc51a9b not found: ID does not exist" Nov 01 21:08:38.237519 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.237470 5633 scope.go:122] "RemoveContainer" containerID="c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd" Nov 01 21:08:38.247126 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.247108 5633 scope.go:122] "RemoveContainer" containerID="c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd" Nov 01 21:08:38.247467 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:08:38.247440 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd\": container with ID starting with c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd not found: ID does not exist" containerID="c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd" Nov 01 21:08:38.247520 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.247477 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd"} err="failed to get container status \"c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd\": rpc error: code = NotFound desc = could not find container \"c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd\": container with ID starting with c58d1949d89ed53d6f9516aaf18891295a91b7dbc8f3e0543d69ba69f751a1bd not found: ID does not exist" Nov 01 21:08:38.247520 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.247494 5633 scope.go:122] "RemoveContainer" containerID="e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a" Nov 01 21:08:38.256310 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.256290 5633 scope.go:122] "RemoveContainer" containerID="e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a" Nov 01 21:08:38.256656 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:08:38.256637 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a\": container with ID starting with e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a not found: ID does not exist" containerID="e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a" Nov 01 21:08:38.256726 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.256663 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a"} err="failed to get container status \"e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a\": rpc error: code = NotFound desc = could not find container \"e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a\": container with ID starting with e92bcd42153bb72c699f06d68d3f6ae63327de50182f85935ce853dedf31b72a not found: ID does not exist" Nov 01 21:08:38.375261 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.375225 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d"] Nov 01 21:08:38.427921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.427824 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8ld9d"] Nov 01 21:08:38.466464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.466428 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" path="/var/lib/kubelet/pods/4a167059-09af-4a4b-a89b-dd8f0f3cde14/volumes" Nov 01 21:08:38.490795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.490762 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq"] Nov 01 21:08:38.495186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.495155 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-hqlkq"] Nov 01 21:08:38.602904 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.602859 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9"] Nov 01 21:08:38.606588 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.606551 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f66n9"] Nov 01 21:08:38.713389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.713297 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn"] Nov 01 21:08:38.716706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:38.716677 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4rtsn"] Nov 01 21:08:39.283217 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:39.283181 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:39.800822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:39.800784 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:40.465647 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:40.465613 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" path="/var/lib/kubelet/pods/72f0b64c-9675-47af-b0d7-185ea0b8afa1/volumes" Nov 01 21:08:40.466065 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:40.466052 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" path="/var/lib/kubelet/pods/a0da0ce9-0ce9-4609-a2c2-b0a6594b3093/volumes" Nov 01 21:08:40.466457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:40.466444 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" path="/var/lib/kubelet/pods/c2bddec4-65c4-4adf-ae35-3894be629d16/volumes" Nov 01 21:08:43.967013 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:43.966979 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:46.156166 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:08:46.156125 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:08:48.737360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:48.737299 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:49.044456 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:49.044356 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v"] Nov 01 21:08:49.093304 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:49.093258 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bf212a-q788v"] Nov 01 21:08:50.466903 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:50.466865 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3702827b-4f17-4269-9d92-45d06aba0846" path="/var/lib/kubelet/pods/3702827b-4f17-4269-9d92-45d06aba0846/volumes" Nov 01 21:08:52.686224 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:52.686182 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:52.852667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:52.852625 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:53.375254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:53.375213 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:08:58.074947 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:08:58.074904 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:01.045266 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:01.045219 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6"] Nov 01 21:09:01.048038 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:01.048013 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9471878a-8cpn6"] Nov 01 21:09:02.466147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:02.466102 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac3edcf-e108-4b3d-b3ae-9d93a7160585" path="/var/lib/kubelet/pods/3ac3edcf-e108-4b3d-b3ae-9d93a7160585/volumes" Nov 01 21:09:02.775117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:02.775020 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:07.470871 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:07.470826 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:07.979405 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:07.979368 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:08.117880 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:08.117832 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:12.079278 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.079231 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:12.109147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109111 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l"] Nov 01 21:09:12.109580 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109566 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" Nov 01 21:09:12.109641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109582 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" Nov 01 21:09:12.109641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109596 5633 cpu_manager.go:414] "containerMap: removing container" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" Nov 01 21:09:12.109641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109601 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" Nov 01 21:09:12.109641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109622 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3ac3edcf-e108-4b3d-b3ae-9d93a7160585" containerName="revoke-keys" Nov 01 21:09:12.109641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109628 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac3edcf-e108-4b3d-b3ae-9d93a7160585" containerName="revoke-keys" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109648 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109653 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109664 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109668 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109731 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac3edcf-e108-4b3d-b3ae-9d93a7160585" containerName="revoke-keys" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109741 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0da0ce9-0ce9-4609-a2c2-b0a6594b3093" containerName="istio-proxy" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109748 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a167059-09af-4a4b-a89b-dd8f0f3cde14" containerName="istio-proxy" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109755 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bddec4-65c4-4adf-ae35-3894be629d16" containerName="istio-proxy" Nov 01 21:09:12.109830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.109760 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f0b64c-9675-47af-b0d7-185ea0b8afa1" containerName="istio-proxy" Nov 01 21:09:12.112971 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.112955 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.168826 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.168796 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l"] Nov 01 21:09:12.214583 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214545 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\"" Nov 01 21:09:12.214583 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214545 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-896aa3063754\"" Nov 01 21:09:12.214583 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214580 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\"" Nov 01 21:09:12.214861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214547 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-14958f7d1994\"" Nov 01 21:09:12.214861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214552 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-dd468d87ff44\"" Nov 01 21:09:12.214861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214546 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\"" Nov 01 21:09:12.214861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214624 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\"" Nov 01 21:09:12.214861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.214705 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-nnqcq\"" Nov 01 21:09:12.217200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.217180 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p57l\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-kube-api-access-2p57l\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.217264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.217229 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.217314 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.217301 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.217392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.217379 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.318555 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.318517 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.318783 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.318572 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.318783 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.318620 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.318783 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.318659 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p57l\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-kube-api-access-2p57l\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.319244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.319219 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.319319 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.319277 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.321410 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.321392 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.374931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.374861 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p57l\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-kube-api-access-2p57l\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.426985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.426942 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:12.692926 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.692826 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:09:12.809082 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:12.809052 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l"] Nov 01 21:09:12.811515 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:09:12.811484 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b05c88_ff97_4e34_becf_1db969bc5036.slice/crio-72e3154627e934d5998c6d1cfa3235ca4a6760d693e3cd2a12f09e85ff0a45ab WatchSource:0}: Error finding container 72e3154627e934d5998c6d1cfa3235ca4a6760d693e3cd2a12f09e85ff0a45ab: Status 404 returned error can't find the container with id 72e3154627e934d5998c6d1cfa3235ca4a6760d693e3cd2a12f09e85ff0a45ab Nov 01 21:09:13.421186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:13.421148 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" event={"ID":"c1b05c88-ff97-4e34-becf-1db969bc5036","Type":"ContainerStarted","Data":"d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3"} Nov 01 21:09:13.421736 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:13.421206 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" event={"ID":"c1b05c88-ff97-4e34-becf-1db969bc5036","Type":"ContainerStarted","Data":"72e3154627e934d5998c6d1cfa3235ca4a6760d693e3cd2a12f09e85ff0a45ab"} Nov 01 21:09:13.421736 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:13.421368 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:13.533581 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:13.533528 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" podStartSLOduration=1.533509423 podStartE2EDuration="1.533509423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:13.532443439 +0000 UTC m=+3961.206702698" watchObservedRunningTime="2026-11-01 21:09:13.533509423 +0000 UTC m=+3961.207768679" Nov 01 21:09:14.609385 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.609352 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb"] Nov 01 21:09:14.613282 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.613261 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.668792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.668759 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb"] Nov 01 21:09:14.714718 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714678 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-a144a099ec31\"" Nov 01 21:09:14.714718 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714671 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-9c12faaff243\"" Nov 01 21:09:14.714718 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714678 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-f1b0d0f19d3d\"" Nov 01 21:09:14.715006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714726 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 21:09:14.715006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714691 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 21:09:14.715006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714691 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 21:09:14.715006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714762 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-zshnb\"" Nov 01 21:09:14.715006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.714736 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 21:09:14.743340 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.743308 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.743536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.743454 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f95g\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-kube-api-access-5f95g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.743536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.743483 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.743536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.743512 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.844145 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.844103 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.844362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.844175 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f95g\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-kube-api-access-5f95g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.844362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.844210 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.844362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.844243 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.845003 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.844982 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.845095 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.845076 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.847010 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.846989 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.900237 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.900161 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f95g\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-kube-api-access-5f95g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:14.929076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:14.929038 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:15.305009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:15.304985 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb"] Nov 01 21:09:15.306651 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:09:15.306625 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a04272_62dc_416a_bae4_631c3ec65e36.slice/crio-497b79d68d965140cb3e303e623be1fdc6d3fe456c2be52dd78b33c66685477f WatchSource:0}: Error finding container 497b79d68d965140cb3e303e623be1fdc6d3fe456c2be52dd78b33c66685477f: Status 404 returned error can't find the container with id 497b79d68d965140cb3e303e623be1fdc6d3fe456c2be52dd78b33c66685477f Nov 01 21:09:15.434760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:15.434729 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" event={"ID":"d5a04272-62dc-416a-bae4-631c3ec65e36","Type":"ContainerStarted","Data":"11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb"} Nov 01 21:09:15.434963 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:15.434776 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" event={"ID":"d5a04272-62dc-416a-bae4-631c3ec65e36","Type":"ContainerStarted","Data":"497b79d68d965140cb3e303e623be1fdc6d3fe456c2be52dd78b33c66685477f"} Nov 01 21:09:15.434963 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:15.434862 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:15.550642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:15.550550 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" podStartSLOduration=1.550535348 podStartE2EDuration="1.550535348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:15.549183217 +0000 UTC m=+3963.223442474" watchObservedRunningTime="2026-11-01 21:09:15.550535348 +0000 UTC m=+3963.224794622" Nov 01 21:09:16.157110 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:09:16.157073 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:09:16.789390 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:16.789358 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:19.787418 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:19.787381 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:20.083659 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:20.083552 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:21.479911 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:21.479821 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:24.430757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:24.430727 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:25.213306 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.213270 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh"] Nov 01 21:09:25.218650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.218626 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.272572 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.272533 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh"] Nov 01 21:09:25.318250 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.318219 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-414b1f\"/\"e2e-usage-sim-9db0ca-cert\"" Nov 01 21:09:25.318250 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.318221 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-414b1f\"/\"kube-root-ca.crt\"" Nov 01 21:09:25.318501 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.318225 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-414b1f\"/\"openshift-service-ca.crt\"" Nov 01 21:09:25.319744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.319726 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-414b1f\"/\"default-dockercfg-9ghm9\"" Nov 01 21:09:25.359504 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.359468 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwxr\" (UniqueName: \"kubernetes.io/projected/da4d06d4-7fa5-4033-999f-fc2873fb505b-kube-api-access-zkwxr\") pod \"e2e-usage-sim-9db0ca-6bd84c9488-h59bh\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.359700 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.359601 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da4d06d4-7fa5-4033-999f-fc2873fb505b-tls\") pod \"e2e-usage-sim-9db0ca-6bd84c9488-h59bh\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.461007 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.460968 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwxr\" (UniqueName: \"kubernetes.io/projected/da4d06d4-7fa5-4033-999f-fc2873fb505b-kube-api-access-zkwxr\") pod \"e2e-usage-sim-9db0ca-6bd84c9488-h59bh\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.461512 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.461075 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da4d06d4-7fa5-4033-999f-fc2873fb505b-tls\") pod \"e2e-usage-sim-9db0ca-6bd84c9488-h59bh\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.463849 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.463795 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da4d06d4-7fa5-4033-999f-fc2873fb505b-tls\") pod \"e2e-usage-sim-9db0ca-6bd84c9488-h59bh\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.517116 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.517078 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwxr\" (UniqueName: \"kubernetes.io/projected/da4d06d4-7fa5-4033-999f-fc2873fb505b-kube-api-access-zkwxr\") pod \"e2e-usage-sim-9db0ca-6bd84c9488-h59bh\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.531971 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.531941 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:25.905898 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:25.905865 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh"] Nov 01 21:09:25.907025 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:09:25.906997 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda4d06d4_7fa5_4033_999f_fc2873fb505b.slice/crio-1aa03cd0f5dbc9144972f0173add1f64ca3441170c158746aff43eb2c7e03f16 WatchSource:0}: Error finding container 1aa03cd0f5dbc9144972f0173add1f64ca3441170c158746aff43eb2c7e03f16: Status 404 returned error can't find the container with id 1aa03cd0f5dbc9144972f0173add1f64ca3441170c158746aff43eb2c7e03f16 Nov 01 21:09:26.092981 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:26.092936 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:26.444548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:26.444516 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:26.498315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:26.498276 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" event={"ID":"da4d06d4-7fa5-4033-999f-fc2873fb505b","Type":"ContainerStarted","Data":"3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739"} Nov 01 21:09:26.498315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:26.498316 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" event={"ID":"da4d06d4-7fa5-4033-999f-fc2873fb505b","Type":"ContainerStarted","Data":"1aa03cd0f5dbc9144972f0173add1f64ca3441170c158746aff43eb2c7e03f16"} Nov 01 21:09:26.498869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:26.498447 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:26.503118 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:26.503095 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:09:26.698711 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:26.698608 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" podStartSLOduration=1.6985917860000002 podStartE2EDuration="1.698591786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:26.696923681 +0000 UTC m=+3974.371182938" watchObservedRunningTime="2026-11-01 21:09:26.698591786 +0000 UTC m=+3974.372851041" Nov 01 21:09:30.817815 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:30.817776 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:31.459094 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:31.459047 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:31.585581 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:31.585545 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:35.408072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:35.408030 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:36.375567 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:36.375527 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:09:40.085325 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:40.085284 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:40.250317 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:40.250283 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:44.757985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:44.757945 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:46.156522 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:09:46.156486 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:09:47.318471 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:47.318435 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb"] Nov 01 21:09:49.333990 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:49.333953 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l"] Nov 01 21:09:49.481271 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:49.481211 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:52.319040 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.318992 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" podUID="d5a04272-62dc-416a-bae4-631c3ec65e36" containerName="payload-processing" containerID="cri-o://11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb" gracePeriod=25 Nov 01 21:09:52.576701 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.576640 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:52.644490 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.644461 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-kube-api-access-5f95g\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-kube-api-access-5f95g\") pod \"d5a04272-62dc-416a-bae4-631c3ec65e36\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " Nov 01 21:09:52.644490 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.644496 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-routing-overlay-volume\") pod \"d5a04272-62dc-416a-bae4-631c3ec65e36\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " Nov 01 21:09:52.644771 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.644568 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-plugins-config-volume\") pod \"d5a04272-62dc-416a-bae4-631c3ec65e36\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " Nov 01 21:09:52.644771 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.644664 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-provider-credentials\") pod \"d5a04272-62dc-416a-bae4-631c3ec65e36\" (UID: \"d5a04272-62dc-416a-bae4-631c3ec65e36\") " Nov 01 21:09:52.644978 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.644939 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-routing-overlay-volume" pod "d5a04272-62dc-416a-bae4-631c3ec65e36" (UID: "d5a04272-62dc-416a-bae4-631c3ec65e36"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:52.645108 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.645019 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-plugins-config-volume" pod "d5a04272-62dc-416a-bae4-631c3ec65e36" (UID: "d5a04272-62dc-416a-bae4-631c3ec65e36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:52.647060 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.647027 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-kube-api-access-5f95g" pod "d5a04272-62dc-416a-bae4-631c3ec65e36" (UID: "d5a04272-62dc-416a-bae4-631c3ec65e36"). InnerVolumeSpecName "kube-api-access-5f95g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:52.647177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.647073 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-provider-credentials" pod "d5a04272-62dc-416a-bae4-631c3ec65e36" (UID: "d5a04272-62dc-416a-bae4-631c3ec65e36"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:52.649948 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.649919 5633 generic.go:356] "Generic (PLEG): container finished" podID="d5a04272-62dc-416a-bae4-631c3ec65e36" containerID="11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb" exitCode=0 Nov 01 21:09:52.650077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.649973 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" Nov 01 21:09:52.650077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.649999 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" event={"ID":"d5a04272-62dc-416a-bae4-631c3ec65e36","Type":"ContainerDied","Data":"11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb"} Nov 01 21:09:52.650077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.650057 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb" event={"ID":"d5a04272-62dc-416a-bae4-631c3ec65e36","Type":"ContainerDied","Data":"497b79d68d965140cb3e303e623be1fdc6d3fe456c2be52dd78b33c66685477f"} Nov 01 21:09:52.650260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.650088 5633 scope.go:122] "RemoveContainer" containerID="11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb" Nov 01 21:09:52.652619 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.652594 5633 generic.go:356] "Generic (PLEG): container finished" podID="e704eaa9-50e3-4234-9357-f374ad06a72a" containerID="6643642df36908f53ff88429637cda6fa766043abbd11f2e486207f2530aaf3a" exitCode=1 Nov 01 21:09:52.652740 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.652664 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" event={"ID":"e704eaa9-50e3-4234-9357-f374ad06a72a","Type":"ContainerDied","Data":"6643642df36908f53ff88429637cda6fa766043abbd11f2e486207f2530aaf3a"} Nov 01 21:09:52.653097 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.653079 5633 scope.go:122] "RemoveContainer" containerID="6643642df36908f53ff88429637cda6fa766043abbd11f2e486207f2530aaf3a" Nov 01 21:09:52.662528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.662507 5633 scope.go:122] "RemoveContainer" containerID="11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb" Nov 01 21:09:52.662800 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:09:52.662782 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb\": container with ID starting with 11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb not found: ID does not exist" containerID="11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb" Nov 01 21:09:52.662847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.662808 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb"} err="failed to get container status \"11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb\": rpc error: code = NotFound desc = could not find container \"11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb\": container with ID starting with 11d74e3b2372b40300cc541cb8f51703b3486284cd4185fd8db6cf821cf862fb not found: ID does not exist" Nov 01 21:09:52.737265 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.737235 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 21:09:52.746251 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.746129 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:52.746251 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.746149 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f95g\" (UniqueName: \"kubernetes.io/projected/d5a04272-62dc-416a-bae4-631c3ec65e36-kube-api-access-5f95g\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:52.746251 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.746160 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:52.746251 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.746169 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a04272-62dc-416a-bae4-631c3ec65e36-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:52.918417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.914445 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb"] Nov 01 21:09:52.924957 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:52.924923 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5l8lbb"] Nov 01 21:09:53.660785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:53.660750 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" event={"ID":"e704eaa9-50e3-4234-9357-f374ad06a72a","Type":"ContainerStarted","Data":"5ed2a8f4e1c429eccb0243edee658573b3332fc676a62604e063d4eeb5ad6a0a"} Nov 01 21:09:53.661268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:53.660968 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 21:09:54.194257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.194214 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:09:54.334314 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.334223 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" podUID="c1b05c88-ff97-4e34-becf-1db969bc5036" containerName="payload-processing" containerID="cri-o://d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3" gracePeriod=25 Nov 01 21:09:54.427701 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:09:54.427671 5633 logging.go:55] [core] [Channel #4909 SubChannel #4910]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.253:50052", ServerName: "10.134.0.253:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.253:50052: connect: connection refused" Nov 01 21:09:54.469084 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.468791 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a04272-62dc-416a-bae4-631c3ec65e36" path="/var/lib/kubelet/pods/d5a04272-62dc-416a-bae4-631c3ec65e36/volumes" Nov 01 21:09:54.608466 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.608436 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:54.669648 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.669620 5633 generic.go:356] "Generic (PLEG): container finished" podID="c1b05c88-ff97-4e34-becf-1db969bc5036" containerID="d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3" exitCode=0 Nov 01 21:09:54.670076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.669673 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" Nov 01 21:09:54.670076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.669711 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" event={"ID":"c1b05c88-ff97-4e34-becf-1db969bc5036","Type":"ContainerDied","Data":"d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3"} Nov 01 21:09:54.670076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.669796 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" event={"ID":"c1b05c88-ff97-4e34-becf-1db969bc5036","Type":"ContainerDied","Data":"72e3154627e934d5998c6d1cfa3235ca4a6760d693e3cd2a12f09e85ff0a45ab"} Nov 01 21:09:54.670076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.669826 5633 scope.go:122] "RemoveContainer" containerID="d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3" Nov 01 21:09:54.679881 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.679861 5633 scope.go:122] "RemoveContainer" containerID="d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3" Nov 01 21:09:54.680219 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:09:54.680193 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3\": container with ID starting with d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3 not found: ID does not exist" containerID="d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3" Nov 01 21:09:54.680277 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.680235 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3"} err="failed to get container status \"d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3\": rpc error: code = NotFound desc = could not find container \"d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3\": container with ID starting with d55a2d3ff9cf5b8c33762f80e7702b0af9fd70e0f69b004ad703f3bdea5ad0c3 not found: ID does not exist" Nov 01 21:09:54.767533 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.767491 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-provider-credentials\") pod \"c1b05c88-ff97-4e34-becf-1db969bc5036\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " Nov 01 21:09:54.767732 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.767566 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-plugins-config-volume\") pod \"c1b05c88-ff97-4e34-becf-1db969bc5036\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " Nov 01 21:09:54.767732 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.767615 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-kube-api-access-2p57l\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-kube-api-access-2p57l\") pod \"c1b05c88-ff97-4e34-becf-1db969bc5036\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " Nov 01 21:09:54.767732 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.767670 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-routing-overlay-volume\") pod \"c1b05c88-ff97-4e34-becf-1db969bc5036\" (UID: \"c1b05c88-ff97-4e34-becf-1db969bc5036\") " Nov 01 21:09:54.768088 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.768056 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-plugins-config-volume" pod "c1b05c88-ff97-4e34-becf-1db969bc5036" (UID: "c1b05c88-ff97-4e34-becf-1db969bc5036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:54.768230 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.768180 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-routing-overlay-volume" pod "c1b05c88-ff97-4e34-becf-1db969bc5036" (UID: "c1b05c88-ff97-4e34-becf-1db969bc5036"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:54.769995 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.769968 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-provider-credentials" pod "c1b05c88-ff97-4e34-becf-1db969bc5036" (UID: "c1b05c88-ff97-4e34-becf-1db969bc5036"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:54.769995 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.769970 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-kube-api-access-2p57l" pod "c1b05c88-ff97-4e34-becf-1db969bc5036" (UID: "c1b05c88-ff97-4e34-becf-1db969bc5036"). InnerVolumeSpecName "kube-api-access-2p57l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:54.869267 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.869237 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:54.869267 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.869265 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:54.869267 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.869274 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p57l\" (UniqueName: \"kubernetes.io/projected/c1b05c88-ff97-4e34-becf-1db969bc5036-kube-api-access-2p57l\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:54.869522 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:54.869283 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c1b05c88-ff97-4e34-becf-1db969bc5036-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:55.089549 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:55.089443 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l"] Nov 01 21:09:55.092062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:55.092039 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l"] Nov 01 21:09:55.427575 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:55.427475 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdf9p2l" podUID="c1b05c88-ff97-4e34-becf-1db969bc5036" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.253:50052\" within 1s: context deadline exceeded" Nov 01 21:09:56.465689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:56.465651 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b05c88-ff97-4e34-becf-1db969bc5036" path="/var/lib/kubelet/pods/c1b05c88-ff97-4e34-becf-1db969bc5036/volumes" Nov 01 21:09:58.890926 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:09:58.890884 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:03.577445 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:03.577399 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:03.725709 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:03.725671 5633 generic.go:356] "Generic (PLEG): container finished" podID="e704eaa9-50e3-4234-9357-f374ad06a72a" containerID="5ed2a8f4e1c429eccb0243edee658573b3332fc676a62604e063d4eeb5ad6a0a" exitCode=1 Nov 01 21:10:03.725895 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:03.725745 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" event={"ID":"e704eaa9-50e3-4234-9357-f374ad06a72a","Type":"ContainerDied","Data":"5ed2a8f4e1c429eccb0243edee658573b3332fc676a62604e063d4eeb5ad6a0a"} Nov 01 21:10:03.725895 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:03.725812 5633 scope.go:122] "RemoveContainer" containerID="6643642df36908f53ff88429637cda6fa766043abbd11f2e486207f2530aaf3a" Nov 01 21:10:03.726190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:03.726173 5633 scope.go:122] "RemoveContainer" containerID="5ed2a8f4e1c429eccb0243edee658573b3332fc676a62604e063d4eeb5ad6a0a" Nov 01 21:10:03.726498 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:10:03.726478 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-8557cbdb84-mlhns_kuadrant-system(e704eaa9-50e3-4234-9357-f374ad06a72a)\"" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" podUID="e704eaa9-50e3-4234-9357-f374ad06a72a" Nov 01 21:10:08.378884 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:08.378836 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:13.090533 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:13.090490 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:13.706137 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:13.706101 5633 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 21:10:13.706554 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:13.706537 5633 scope.go:122] "RemoveContainer" containerID="5ed2a8f4e1c429eccb0243edee658573b3332fc676a62604e063d4eeb5ad6a0a" Nov 01 21:10:13.797118 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:13.797080 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" event={"ID":"e704eaa9-50e3-4234-9357-f374ad06a72a","Type":"ContainerStarted","Data":"f27968c6825391c4f0b8a0cbbf6c4adc12420faa14a3901f814f22906d53f908"} Nov 01 21:10:13.797369 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:13.797351 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 21:10:16.156651 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:10:16.156613 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:10:17.180318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:17.180284 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:10:17.774354 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:17.774293 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:18.294453 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:18.294358 5633 scope.go:122] "RemoveContainer" containerID="b062c05932d3b548a997e650b91c3fafd5aaf058a0d532e97c75ee080ee48e63" Nov 01 21:10:18.303640 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:18.303615 5633 scope.go:122] "RemoveContainer" containerID="22311c95b9e495965f79a9e26a5f54193fce5a1980e89d8982979bf534b68ee3" Nov 01 21:10:22.497865 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:22.497818 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:24.805702 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:24.805672 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mlhns" Nov 01 21:10:27.271649 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:27.271602 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:30.782508 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:30.782476 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:10:31.889149 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:31.889114 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:36.573847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:36.573800 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:40.483138 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:40.483100 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:41.287390 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:41.287347 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:45.913089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:45.913044 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:46.161184 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:10:46.161143 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:10:50.573343 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:50.573285 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:55.273284 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:55.273183 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:10:59.896021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:10:59.895984 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:04.599209 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:04.599156 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:09.291436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:09.291388 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:13.975168 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:13.975125 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:16.162911 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:11:16.162863 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:11:18.684180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:18.684132 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:23.385321 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:23.385284 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:27.996684 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:27.996638 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:32.679352 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:32.679281 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:37.372981 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:37.372945 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:40.823489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:40.823454 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:42.074880 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:42.074840 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:46.156875 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:11:46.156834 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:11:46.762801 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:46.762765 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:51.478203 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:51.478166 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:11:56.166846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:11:56.166810 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:00.783658 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:00.783613 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:05.476150 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:05.476110 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:10.168118 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:10.168075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:14.865942 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:14.865896 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:16.156770 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:12:16.156727 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:12:19.583352 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:19.583299 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:24.273505 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:24.273415 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:28.893032 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:28.892981 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:33.588200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:33.588158 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:36.444356 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:36.444291 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:36.634057 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:36.634016 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:38.277133 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:38.277100 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:41.170269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:41.170229 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:42.964072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:42.964035 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:46.166876 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:12:46.166832 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:12:47.690233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:47.690194 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:52.376629 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:52.376590 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:12:57.068725 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:12:57.068687 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:01.779112 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:01.779074 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:06.476250 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:06.476198 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:11.193402 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:11.193362 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:15.888014 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:15.887971 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:16.156784 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:13:16.156701 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:13:17.346819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:17.346781 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:20.585044 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:20.585001 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:25.274569 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:25.274534 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:29.992400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:29.992356 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:34.722501 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:34.722455 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:39.353827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.353790 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9"] Nov 01 21:13:39.354276 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.354262 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c1b05c88-ff97-4e34-becf-1db969bc5036" containerName="payload-processing" Nov 01 21:13:39.354360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.354277 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b05c88-ff97-4e34-becf-1db969bc5036" containerName="payload-processing" Nov 01 21:13:39.354360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.354288 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d5a04272-62dc-416a-bae4-631c3ec65e36" containerName="payload-processing" Nov 01 21:13:39.354360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.354293 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a04272-62dc-416a-bae4-631c3ec65e36" containerName="payload-processing" Nov 01 21:13:39.354480 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.354380 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b05c88-ff97-4e34-becf-1db969bc5036" containerName="payload-processing" Nov 01 21:13:39.354480 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.354388 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a04272-62dc-416a-bae4-631c3ec65e36" containerName="payload-processing" Nov 01 21:13:39.357696 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.357680 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.384318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.384275 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:39.387408 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.387384 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.387511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.387422 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.387559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.387530 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmmc\" (UniqueName: \"kubernetes.io/projected/76fade02-8ea2-4e45-b384-58b26246797f-kube-api-access-rrmmc\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.409548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.409512 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:13:39.458811 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.458780 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:13:39.458811 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.458781 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:13:39.459037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.458787 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:13:39.461708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.461684 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9"] Nov 01 21:13:39.488253 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.488220 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmmc\" (UniqueName: \"kubernetes.io/projected/76fade02-8ea2-4e45-b384-58b26246797f-kube-api-access-rrmmc\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.488466 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.488325 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.488466 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.488393 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.488990 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.488967 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.490889 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.490868 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.544239 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.544203 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmmc\" (UniqueName: \"kubernetes.io/projected/76fade02-8ea2-4e45-b384-58b26246797f-kube-api-access-rrmmc\") pod \"maas-api-revoke-sub-63bb825d-tg8d9\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:39.671023 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:39.670933 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:40.044493 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:13:40.044463 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fade02_8ea2_4e45_b384_58b26246797f.slice/crio-a35a1819276c969d00060a67a10c9431d20462b43a49f4e4e288d12c3aaa551e WatchSource:0}: Error finding container a35a1819276c969d00060a67a10c9431d20462b43a49f4e4e288d12c3aaa551e: Status 404 returned error can't find the container with id a35a1819276c969d00060a67a10c9431d20462b43a49f4e4e288d12c3aaa551e Nov 01 21:13:40.045081 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:40.045051 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9"] Nov 01 21:13:40.931498 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:40.931461 5633 generic.go:356] "Generic (PLEG): container finished" podID="76fade02-8ea2-4e45-b384-58b26246797f" containerID="546a191410ae8ed9444ad211e827ecdf55d5dd53566cad09078d73fea0328f98" exitCode=0 Nov 01 21:13:40.931936 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:40.931536 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" event={"ID":"76fade02-8ea2-4e45-b384-58b26246797f","Type":"ContainerDied","Data":"546a191410ae8ed9444ad211e827ecdf55d5dd53566cad09078d73fea0328f98"} Nov 01 21:13:40.931936 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:40.931590 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" event={"ID":"76fade02-8ea2-4e45-b384-58b26246797f","Type":"ContainerStarted","Data":"a35a1819276c969d00060a67a10c9431d20462b43a49f4e4e288d12c3aaa551e"} Nov 01 21:13:41.511183 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:41.511152 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:42.088869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.088843 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:42.115912 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.115878 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fade02-8ea2-4e45-b384-58b26246797f-kube-api-access-rrmmc\" (UniqueName: \"kubernetes.io/projected/76fade02-8ea2-4e45-b384-58b26246797f-kube-api-access-rrmmc\") pod \"76fade02-8ea2-4e45-b384-58b26246797f\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " Nov 01 21:13:42.116085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.115937 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-openshift-ca\") pod \"76fade02-8ea2-4e45-b384-58b26246797f\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " Nov 01 21:13:42.116085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.116033 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-xks-ca\") pod \"76fade02-8ea2-4e45-b384-58b26246797f\" (UID: \"76fade02-8ea2-4e45-b384-58b26246797f\") " Nov 01 21:13:42.116303 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.116279 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-openshift-ca" pod "76fade02-8ea2-4e45-b384-58b26246797f" (UID: "76fade02-8ea2-4e45-b384-58b26246797f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:42.118180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.118154 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-xks-ca" pod "76fade02-8ea2-4e45-b384-58b26246797f" (UID: "76fade02-8ea2-4e45-b384-58b26246797f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:42.118180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.118173 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fade02-8ea2-4e45-b384-58b26246797f-kube-api-access-rrmmc" pod "76fade02-8ea2-4e45-b384-58b26246797f" (UID: "76fade02-8ea2-4e45-b384-58b26246797f"). InnerVolumeSpecName "kube-api-access-rrmmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:42.217216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.217175 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmmc\" (UniqueName: \"kubernetes.io/projected/76fade02-8ea2-4e45-b384-58b26246797f-kube-api-access-rrmmc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:42.217216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.217208 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:42.217216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.217219 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fade02-8ea2-4e45-b384-58b26246797f-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:42.944630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.944540 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" Nov 01 21:13:42.944630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.944579 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9" event={"ID":"76fade02-8ea2-4e45-b384-58b26246797f","Type":"ContainerDied","Data":"a35a1819276c969d00060a67a10c9431d20462b43a49f4e4e288d12c3aaa551e"} Nov 01 21:13:42.944827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:42.944632 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35a1819276c969d00060a67a10c9431d20462b43a49f4e4e288d12c3aaa551e" Nov 01 21:13:44.086709 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:44.086668 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:46.155936 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:13:46.155899 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:13:48.788765 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:48.788724 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:51.059018 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.058935 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv"] Nov 01 21:13:51.059445 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.059435 5633 cpu_manager.go:414] "containerMap: removing container" podUID="76fade02-8ea2-4e45-b384-58b26246797f" containerName="revoke-keys" Nov 01 21:13:51.059502 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.059448 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fade02-8ea2-4e45-b384-58b26246797f" containerName="revoke-keys" Nov 01 21:13:51.059548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.059532 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fade02-8ea2-4e45-b384-58b26246797f" containerName="revoke-keys" Nov 01 21:13:51.062749 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.062731 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.108739 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.108700 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.108932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.108751 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7hkp\" (UniqueName: \"kubernetes.io/projected/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-kube-api-access-j7hkp\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.108932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.108818 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.166096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.166054 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:13:51.166297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.166103 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:13:51.166297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.166135 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:13:51.166297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.166158 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:13:51.169080 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.169058 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv"] Nov 01 21:13:51.209634 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.209599 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7hkp\" (UniqueName: \"kubernetes.io/projected/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-kube-api-access-j7hkp\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.209830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.209643 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.209830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.209737 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.210422 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.210395 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.212272 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.212251 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.265630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.265591 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7hkp\" (UniqueName: \"kubernetes.io/projected/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-kube-api-access-j7hkp\") pod \"maas-api-revoke-sub-615eb5af-gtlrv\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.376302 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.376215 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:51.750740 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.750705 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv"] Nov 01 21:13:51.751388 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:13:51.751356 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1722d9d1_e4ae_41a4_b9eb_afa33dab4de0.slice/crio-58e7f822560223c39e7f19988730af70d5f683b0563eb2fb99a9f56eb9b54993 WatchSource:0}: Error finding container 58e7f822560223c39e7f19988730af70d5f683b0563eb2fb99a9f56eb9b54993: Status 404 returned error can't find the container with id 58e7f822560223c39e7f19988730af70d5f683b0563eb2fb99a9f56eb9b54993 Nov 01 21:13:51.997021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:51.995509 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" event={"ID":"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0","Type":"ContainerStarted","Data":"58e7f822560223c39e7f19988730af70d5f683b0563eb2fb99a9f56eb9b54993"} Nov 01 21:13:52.777561 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:52.777525 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:13:53.003026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:53.002991 5633 generic.go:356] "Generic (PLEG): container finished" podID="1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" containerID="d44443e3550da26b5d5a4959c637fb339ca6fe45d7a10154f8720a99fbf3145e" exitCode=0 Nov 01 21:13:53.003243 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:53.003070 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" event={"ID":"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0","Type":"ContainerDied","Data":"d44443e3550da26b5d5a4959c637fb339ca6fe45d7a10154f8720a99fbf3145e"} Nov 01 21:13:53.481281 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:53.481249 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:13:54.148313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.148280 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:54.239886 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.239852 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-openshift-ca\") pod \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " Nov 01 21:13:54.240081 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.239957 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-xks-ca\") pod \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " Nov 01 21:13:54.240081 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.240018 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-kube-api-access-j7hkp\" (UniqueName: \"kubernetes.io/projected/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-kube-api-access-j7hkp\") pod \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\" (UID: \"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0\") " Nov 01 21:13:54.240232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.240208 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-openshift-ca" pod "1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" (UID: "1722d9d1-e4ae-41a4-b9eb-afa33dab4de0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:13:54.240315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.240302 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:54.242238 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.242216 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-kube-api-access-j7hkp" pod "1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" (UID: "1722d9d1-e4ae-41a4-b9eb-afa33dab4de0"). InnerVolumeSpecName "kube-api-access-j7hkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:13:54.242323 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.242309 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-xks-ca" pod "1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" (UID: "1722d9d1-e4ae-41a4-b9eb-afa33dab4de0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:13:54.341858 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.341762 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:54.341858 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:54.341800 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7hkp\" (UniqueName: \"kubernetes.io/projected/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0-kube-api-access-j7hkp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:13:55.016543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:55.016505 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" Nov 01 21:13:55.016543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:55.016530 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv" event={"ID":"1722d9d1-e4ae-41a4-b9eb-afa33dab4de0","Type":"ContainerDied","Data":"58e7f822560223c39e7f19988730af70d5f683b0563eb2fb99a9f56eb9b54993"} Nov 01 21:13:55.016766 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:55.016581 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58e7f822560223c39e7f19988730af70d5f683b0563eb2fb99a9f56eb9b54993" Nov 01 21:13:58.178757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:13:58.178719 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:02.975841 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:02.975801 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:04.096880 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:04.096845 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:04.226804 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:04.226766 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:04.757346 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:04.757305 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh"] Nov 01 21:14:04.757556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:04.757534 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" podUID="da4d06d4-7fa5-4033-999f-fc2873fb505b" containerName="sim" containerID="cri-o://3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739" gracePeriod=30 Nov 01 21:14:04.938275 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:04.938240 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:05.021937 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.021870 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:14:05.077908 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.077871 5633 generic.go:356] "Generic (PLEG): container finished" podID="da4d06d4-7fa5-4033-999f-fc2873fb505b" containerID="3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739" exitCode=0 Nov 01 21:14:05.078115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.077920 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" Nov 01 21:14:05.078115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.077977 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" event={"ID":"da4d06d4-7fa5-4033-999f-fc2873fb505b","Type":"ContainerDied","Data":"3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739"} Nov 01 21:14:05.078115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.078017 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh" event={"ID":"da4d06d4-7fa5-4033-999f-fc2873fb505b","Type":"ContainerDied","Data":"1aa03cd0f5dbc9144972f0173add1f64ca3441170c158746aff43eb2c7e03f16"} Nov 01 21:14:05.078115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.078037 5633 scope.go:122] "RemoveContainer" containerID="3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739" Nov 01 21:14:05.088743 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.088698 5633 scope.go:122] "RemoveContainer" containerID="3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739" Nov 01 21:14:05.089002 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:05.088975 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739\": container with ID starting with 3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739 not found: ID does not exist" containerID="3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739" Nov 01 21:14:05.089062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.089016 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739"} err="failed to get container status \"3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739\": rpc error: code = NotFound desc = could not find container \"3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739\": container with ID starting with 3455320147ac9d77d0223d620649a527d303d1fdf5750ec0feadc15487f3d739 not found: ID does not exist" Nov 01 21:14:05.151017 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.150985 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4d06d4-7fa5-4033-999f-fc2873fb505b-tls\" (UniqueName: \"kubernetes.io/secret/da4d06d4-7fa5-4033-999f-fc2873fb505b-tls\") pod \"da4d06d4-7fa5-4033-999f-fc2873fb505b\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " Nov 01 21:14:05.151514 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.151046 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4d06d4-7fa5-4033-999f-fc2873fb505b-kube-api-access-zkwxr\" (UniqueName: \"kubernetes.io/projected/da4d06d4-7fa5-4033-999f-fc2873fb505b-kube-api-access-zkwxr\") pod \"da4d06d4-7fa5-4033-999f-fc2873fb505b\" (UID: \"da4d06d4-7fa5-4033-999f-fc2873fb505b\") " Nov 01 21:14:05.153528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.153502 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4d06d4-7fa5-4033-999f-fc2873fb505b-kube-api-access-zkwxr" pod "da4d06d4-7fa5-4033-999f-fc2873fb505b" (UID: "da4d06d4-7fa5-4033-999f-fc2873fb505b"). InnerVolumeSpecName "kube-api-access-zkwxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:05.153624 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.153581 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4d06d4-7fa5-4033-999f-fc2873fb505b-tls" pod "da4d06d4-7fa5-4033-999f-fc2873fb505b" (UID: "da4d06d4-7fa5-4033-999f-fc2873fb505b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:05.252875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.252839 5633 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da4d06d4-7fa5-4033-999f-fc2873fb505b-tls\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:05.252875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.252870 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkwxr\" (UniqueName: \"kubernetes.io/projected/da4d06d4-7fa5-4033-999f-fc2873fb505b-kube-api-access-zkwxr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:05.497103 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.497070 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh"] Nov 01 21:14:05.499713 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:05.499689 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-414b1f/e2e-usage-sim-9db0ca-6bd84c9488-h59bh"] Nov 01 21:14:06.466858 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:06.466822 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4d06d4-7fa5-4033-999f-fc2873fb505b" path="/var/lib/kubelet/pods/da4d06d4-7fa5-4033-999f-fc2873fb505b/volumes" Nov 01 21:14:07.407819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.407787 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq"] Nov 01 21:14:07.408383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.408317 5633 cpu_manager.go:414] "containerMap: removing container" podUID="da4d06d4-7fa5-4033-999f-fc2873fb505b" containerName="sim" Nov 01 21:14:07.408383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.408348 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4d06d4-7fa5-4033-999f-fc2873fb505b" containerName="sim" Nov 01 21:14:07.408383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.408359 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" containerName="revoke-keys" Nov 01 21:14:07.408383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.408365 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" containerName="revoke-keys" Nov 01 21:14:07.408571 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.408424 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4d06d4-7fa5-4033-999f-fc2873fb505b" containerName="sim" Nov 01 21:14:07.408571 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.408441 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" containerName="revoke-keys" Nov 01 21:14:07.412930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.412909 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.467670 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.467640 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq"] Nov 01 21:14:07.513546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.513497 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-dd468d87ff44\"" Nov 01 21:14:07.513546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.513502 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\"" Nov 01 21:14:07.513546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.513503 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\"" Nov 01 21:14:07.513813 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.513509 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\"" Nov 01 21:14:07.514464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.514451 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-14958f7d1994\"" Nov 01 21:14:07.514547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.514460 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-nnqcq\"" Nov 01 21:14:07.514547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.514470 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-896aa3063754\"" Nov 01 21:14:07.514547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.514486 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\"" Nov 01 21:14:07.586031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.585998 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.586031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.586033 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.586305 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.586137 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.586305 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.586213 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kgln\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-kube-api-access-2kgln\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.598402 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.598364 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:07.687604 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.687503 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.687604 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.687551 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.687864 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.687614 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.687864 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.687647 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgln\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-kube-api-access-2kgln\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.688232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.688208 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.688539 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.688279 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.690314 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.690293 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:07.743958 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:07.743914 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgln\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-kube-api-access-2kgln\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:08.026366 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:08.026300 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:08.202290 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:08.202249 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:14:08.402504 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:08.402470 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq"] Nov 01 21:14:08.403140 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:14:08.403114 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e95b74_bf3b_4e16_b743_16e5a39bd2c0.slice/crio-c15c6189cfc6b2bd69e93c66be171d6dbb9571fd5d4a4178a8ac6be2dd4d805f WatchSource:0}: Error finding container c15c6189cfc6b2bd69e93c66be171d6dbb9571fd5d4a4178a8ac6be2dd4d805f: Status 404 returned error can't find the container with id c15c6189cfc6b2bd69e93c66be171d6dbb9571fd5d4a4178a8ac6be2dd4d805f Nov 01 21:14:09.108510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.108467 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" event={"ID":"83e95b74-bf3b-4e16-b743-16e5a39bd2c0","Type":"ContainerStarted","Data":"89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05"} Nov 01 21:14:09.108921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.108523 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" event={"ID":"83e95b74-bf3b-4e16-b743-16e5a39bd2c0","Type":"ContainerStarted","Data":"c15c6189cfc6b2bd69e93c66be171d6dbb9571fd5d4a4178a8ac6be2dd4d805f"} Nov 01 21:14:09.108921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.108559 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:09.221400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.221303 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" podStartSLOduration=2.2212867259999998 podStartE2EDuration="2.221286726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:09.219824195 +0000 UTC m=+4256.894083451" watchObservedRunningTime="2026-11-01 21:14:09.221286726 +0000 UTC m=+4256.895545980" Nov 01 21:14:09.588485 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.588444 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:09.707521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.707480 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:09.909067 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.908971 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8"] Nov 01 21:14:09.912982 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.912956 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:09.964840 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.964800 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-a144a099ec31\"" Nov 01 21:14:09.965045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.964876 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-zshnb\"" Nov 01 21:14:09.965045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:09.964890 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-9c12faaff243\"" Nov 01 21:14:10.011756 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.011711 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.011944 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.011857 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.011944 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.011916 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.012034 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.011976 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs99g\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-kube-api-access-vs99g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.014077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.014059 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 21:14:10.014181 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.014063 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 21:14:10.014181 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.014064 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 21:14:10.014181 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.014076 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 21:14:10.017183 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.017161 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8"] Nov 01 21:14:10.113073 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.113040 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.113534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.113081 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.113534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.113140 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs99g\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-kube-api-access-vs99g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.113534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.113173 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.113822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.113798 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.113987 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.113966 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.116189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.116164 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.169393 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.169290 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs99g\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-kube-api-access-vs99g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.214313 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:10.214278 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found Nov 01 21:14:10.214313 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:10.214307 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq: secret "agc-cred-896aa3063754" not found Nov 01 21:14:10.214566 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:10.214394 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials podName:83e95b74-bf3b-4e16-b743-16e5a39bd2c0 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:10.714377025 +0000 UTC m=+4258.388636257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0") : secret "agc-cred-896aa3063754" not found Nov 01 21:14:10.227080 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.227052 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:10.379839 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.379801 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7"] Nov 01 21:14:10.384776 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.384750 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.416364 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.416292 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.416364 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.416326 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.416585 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.416480 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jpdf\" (UniqueName: \"kubernetes.io/projected/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-kube-api-access-9jpdf\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.437049 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.436741 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:10.437049 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.436887 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:14:10.437049 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.436895 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:10.437049 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.436949 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:10.439653 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.439626 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7"] Nov 01 21:14:10.518035 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.517985 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jpdf\" (UniqueName: \"kubernetes.io/projected/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-kube-api-access-9jpdf\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.518254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.518136 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.518254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.518157 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.518815 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.518780 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.520690 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.520671 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.573945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.573914 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jpdf\" (UniqueName: \"kubernetes.io/projected/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-kube-api-access-9jpdf\") pod \"maas-api-revoke-sub-f82128b9-fj4m7\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.647818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.647795 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8"] Nov 01 21:14:10.648756 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:14:10.648726 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50b6ce8_d8c7_4a93_bbc2_4791af69efb4.slice/crio-6500103a60ddb20cf3e06a74a70fa31572fde106928ee91e17d041c5c664a278 WatchSource:0}: Error finding container 6500103a60ddb20cf3e06a74a70fa31572fde106928ee91e17d041c5c664a278: Status 404 returned error can't find the container with id 6500103a60ddb20cf3e06a74a70fa31572fde106928ee91e17d041c5c664a278 Nov 01 21:14:10.698791 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:10.698763 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:10.720756 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:10.720725 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found Nov 01 21:14:10.720756 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:10.720756 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq: secret "agc-cred-896aa3063754" not found Nov 01 21:14:10.720954 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:10.720815 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials podName:83e95b74-bf3b-4e16-b743-16e5a39bd2c0 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:11.720798862 +0000 UTC m=+4259.395058111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0") : secret "agc-cred-896aa3063754" not found Nov 01 21:14:11.072692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:11.072663 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7"] Nov 01 21:14:11.073624 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:14:11.073592 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1d20fa_dbc8_4d55_9e86_1cdc23d12ff5.slice/crio-9f0f2840a675e6b8462dadb563953bc628f253b0beaed72a49ce57fc78b68dba WatchSource:0}: Error finding container 9f0f2840a675e6b8462dadb563953bc628f253b0beaed72a49ce57fc78b68dba: Status 404 returned error can't find the container with id 9f0f2840a675e6b8462dadb563953bc628f253b0beaed72a49ce57fc78b68dba Nov 01 21:14:11.128723 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:11.128676 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" event={"ID":"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5","Type":"ContainerStarted","Data":"9f0f2840a675e6b8462dadb563953bc628f253b0beaed72a49ce57fc78b68dba"} Nov 01 21:14:11.132411 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:11.132380 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" event={"ID":"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4","Type":"ContainerStarted","Data":"6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141"} Nov 01 21:14:11.132651 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:11.132439 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" event={"ID":"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4","Type":"ContainerStarted","Data":"6500103a60ddb20cf3e06a74a70fa31572fde106928ee91e17d041c5c664a278"} Nov 01 21:14:11.132651 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:11.132561 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:11.246431 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:11.246359 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" podStartSLOduration=2.246321427 podStartE2EDuration="2.246321427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:11.245401657 +0000 UTC m=+4258.919660928" watchObservedRunningTime="2026-11-01 21:14:11.246321427 +0000 UTC m=+4258.920580719" Nov 01 21:14:11.733591 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:11.733556 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found Nov 01 21:14:11.733591 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:11.733587 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq: secret "agc-cred-896aa3063754" not found Nov 01 21:14:11.733907 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:11.733654 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials podName:83e95b74-bf3b-4e16-b743-16e5a39bd2c0 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:13.733638415 +0000 UTC m=+4261.407897661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0") : secret "agc-cred-896aa3063754" not found Nov 01 21:14:12.139601 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:12.139567 5633 generic.go:356] "Generic (PLEG): container finished" podID="8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" containerID="1b35361b344094e89ef66f65d57ff1c40cb7cff1d15e5ca62149a98a8fbad06c" exitCode=0 Nov 01 21:14:12.140041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:12.139650 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" event={"ID":"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5","Type":"ContainerDied","Data":"1b35361b344094e89ef66f65d57ff1c40cb7cff1d15e5ca62149a98a8fbad06c"} Nov 01 21:14:12.281359 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:12.281220 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:12.358064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:12.358027 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq"] Nov 01 21:14:13.279067 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.279032 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:13.347521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.347482 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-kube-api-access-9jpdf\" (UniqueName: \"kubernetes.io/projected/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-kube-api-access-9jpdf\") pod \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " Nov 01 21:14:13.347703 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.347610 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-openshift-ca\") pod \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " Nov 01 21:14:13.347752 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.347711 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-xks-ca\") pod \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\" (UID: \"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5\") " Nov 01 21:14:13.347955 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.347930 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-openshift-ca" pod "8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" (UID: "8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:13.348096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.348071 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:13.349824 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.349799 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-xks-ca" pod "8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" (UID: "8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:13.349922 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.349908 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-kube-api-access-9jpdf" pod "8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" (UID: "8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5"). InnerVolumeSpecName "kube-api-access-9jpdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:13.449610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.449528 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:13.449610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:13.449558 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jpdf\" (UniqueName: \"kubernetes.io/projected/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5-kube-api-access-9jpdf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:13.753417 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:13.753375 5633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found Nov 01 21:14:13.753417 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:13.753408 5633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq: secret "agc-cred-896aa3063754" not found Nov 01 21:14:13.753683 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:13.753468 5633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials podName:83e95b74-bf3b-4e16-b743-16e5a39bd2c0 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:17.753448778 +0000 UTC m=+4265.427708012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0") : secret "agc-cred-896aa3063754" not found Nov 01 21:14:14.151949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:14.151862 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" Nov 01 21:14:14.151949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:14.151890 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7" event={"ID":"8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5","Type":"ContainerDied","Data":"9f0f2840a675e6b8462dadb563953bc628f253b0beaed72a49ce57fc78b68dba"} Nov 01 21:14:14.151949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:14.151940 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f0f2840a675e6b8462dadb563953bc628f253b0beaed72a49ce57fc78b68dba" Nov 01 21:14:16.157049 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:16.157013 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:14:16.983295 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:16.983253 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:17.359165 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.359030 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" podUID="83e95b74-bf3b-4e16-b743-16e5a39bd2c0" containerName="payload-processing" containerID="cri-o://89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05" gracePeriod=25 Nov 01 21:14:17.622423 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.622324 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:17.693846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.693810 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-plugins-config-volume\") pod \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " Nov 01 21:14:17.693846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.693850 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-kube-api-access-2kgln\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-kube-api-access-2kgln\") pod \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " Nov 01 21:14:17.694077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.693882 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials\") pod \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " Nov 01 21:14:17.694077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.693919 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-routing-overlay-volume\") pod \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\" (UID: \"83e95b74-bf3b-4e16-b743-16e5a39bd2c0\") " Nov 01 21:14:17.694324 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.694300 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-plugins-config-volume" pod "83e95b74-bf3b-4e16-b743-16e5a39bd2c0" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:17.694409 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.694305 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-routing-overlay-volume" pod "83e95b74-bf3b-4e16-b743-16e5a39bd2c0" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:17.696095 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.696072 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-kube-api-access-2kgln" pod "83e95b74-bf3b-4e16-b743-16e5a39bd2c0" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0"). InnerVolumeSpecName "kube-api-access-2kgln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:17.696259 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.696246 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials" pod "83e95b74-bf3b-4e16-b743-16e5a39bd2c0" (UID: "83e95b74-bf3b-4e16-b743-16e5a39bd2c0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:17.795590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.795558 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:17.795590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.795587 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kgln\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-kube-api-access-2kgln\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:17.795590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.795596 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:17.795851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:17.795605 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/83e95b74-bf3b-4e16-b743-16e5a39bd2c0-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:18.174861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.174827 5633 generic.go:356] "Generic (PLEG): container finished" podID="83e95b74-bf3b-4e16-b743-16e5a39bd2c0" containerID="89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05" exitCode=0 Nov 01 21:14:18.175056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.174869 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" event={"ID":"83e95b74-bf3b-4e16-b743-16e5a39bd2c0","Type":"ContainerDied","Data":"89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05"} Nov 01 21:14:18.175056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.174876 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" Nov 01 21:14:18.175056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.174908 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq" event={"ID":"83e95b74-bf3b-4e16-b743-16e5a39bd2c0","Type":"ContainerDied","Data":"c15c6189cfc6b2bd69e93c66be171d6dbb9571fd5d4a4178a8ac6be2dd4d805f"} Nov 01 21:14:18.175056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.174931 5633 scope.go:122] "RemoveContainer" containerID="89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05" Nov 01 21:14:18.185556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.185532 5633 scope.go:122] "RemoveContainer" containerID="89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05" Nov 01 21:14:18.185885 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:18.185866 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05\": container with ID starting with 89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05 not found: ID does not exist" containerID="89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05" Nov 01 21:14:18.185938 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.185894 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05"} err="failed to get container status \"89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05\": rpc error: code = NotFound desc = could not find container \"89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05\": container with ID starting with 89f6f5f10d70924f766f83e16f5d792a1839a36d54838ddb6341cc701dd57f05 not found: ID does not exist" Nov 01 21:14:18.337725 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.337683 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq"] Nov 01 21:14:18.386868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.386837 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-c4bdtghwq"] Nov 01 21:14:18.465675 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:18.465643 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e95b74-bf3b-4e16-b743-16e5a39bd2c0" path="/var/lib/kubelet/pods/83e95b74-bf3b-4e16-b743-16e5a39bd2c0/volumes" Nov 01 21:14:21.255748 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.255705 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:14:21.696822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.696733 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:21.982559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.982525 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn"] Nov 01 21:14:21.983003 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.982990 5633 cpu_manager.go:414] "containerMap: removing container" podUID="8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" containerName="revoke-keys" Nov 01 21:14:21.983066 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.983006 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" containerName="revoke-keys" Nov 01 21:14:21.983066 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.983038 5633 cpu_manager.go:414] "containerMap: removing container" podUID="83e95b74-bf3b-4e16-b743-16e5a39bd2c0" containerName="payload-processing" Nov 01 21:14:21.983066 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.983043 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e95b74-bf3b-4e16-b743-16e5a39bd2c0" containerName="payload-processing" Nov 01 21:14:21.983175 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.983121 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" containerName="revoke-keys" Nov 01 21:14:21.983175 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.983133 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e95b74-bf3b-4e16-b743-16e5a39bd2c0" containerName="payload-processing" Nov 01 21:14:21.986297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:21.986281 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.036964 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.036932 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.036964 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.036975 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.037190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.037131 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmq2\" (UniqueName: \"kubernetes.io/projected/5edda2ac-5594-45c2-b9b4-ec86240b82cc-kube-api-access-7kmq2\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.041929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.041901 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn"] Nov 01 21:14:22.087612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.087576 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:22.087612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.087603 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:22.087860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.087633 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:22.087860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.087690 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:14:22.138177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.138140 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmq2\" (UniqueName: \"kubernetes.io/projected/5edda2ac-5594-45c2-b9b4-ec86240b82cc-kube-api-access-7kmq2\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.138440 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.138222 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.138440 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.138258 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.138909 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.138886 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.140979 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.140953 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.143081 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.143046 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:22.194221 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.194194 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmq2\" (UniqueName: \"kubernetes.io/projected/5edda2ac-5594-45c2-b9b4-ec86240b82cc-kube-api-access-7kmq2\") pod \"maas-api-revoke-sub-09e076c6-v2jhn\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.299274 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.299176 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:22.670244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:22.670207 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn"] Nov 01 21:14:22.671685 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:14:22.671646 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edda2ac_5594_45c2_b9b4_ec86240b82cc.slice/crio-058ade50a3cab6e30a307a458d1861dc4f59db71863212dc4cdb17b307a8d003 WatchSource:0}: Error finding container 058ade50a3cab6e30a307a458d1861dc4f59db71863212dc4cdb17b307a8d003: Status 404 returned error can't find the container with id 058ade50a3cab6e30a307a458d1861dc4f59db71863212dc4cdb17b307a8d003 Nov 01 21:14:23.205827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:23.205787 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" event={"ID":"5edda2ac-5594-45c2-b9b4-ec86240b82cc","Type":"ContainerStarted","Data":"2c54ab6d50a546235a77a1b11a91da1ae0536a7e550ddeed130b07773144cefd"} Nov 01 21:14:23.206015 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:23.205855 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" event={"ID":"5edda2ac-5594-45c2-b9b4-ec86240b82cc","Type":"ContainerStarted","Data":"058ade50a3cab6e30a307a458d1861dc4f59db71863212dc4cdb17b307a8d003"} Nov 01 21:14:23.320538 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:23.320431 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" podStartSLOduration=2.320408246 podStartE2EDuration="2.320408246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:23.316954823 +0000 UTC m=+4270.991214077" watchObservedRunningTime="2026-11-01 21:14:23.320408246 +0000 UTC m=+4270.994667500" Nov 01 21:14:24.212741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:24.212704 5633 generic.go:356] "Generic (PLEG): container finished" podID="5edda2ac-5594-45c2-b9b4-ec86240b82cc" containerID="2c54ab6d50a546235a77a1b11a91da1ae0536a7e550ddeed130b07773144cefd" exitCode=0 Nov 01 21:14:24.212960 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:24.212792 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" event={"ID":"5edda2ac-5594-45c2-b9b4-ec86240b82cc","Type":"ContainerDied","Data":"2c54ab6d50a546235a77a1b11a91da1ae0536a7e550ddeed130b07773144cefd"} Nov 01 21:14:25.365798 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.365771 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:25.474318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.474222 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edda2ac-5594-45c2-b9b4-ec86240b82cc-kube-api-access-7kmq2\" (UniqueName: \"kubernetes.io/projected/5edda2ac-5594-45c2-b9b4-ec86240b82cc-kube-api-access-7kmq2\") pod \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " Nov 01 21:14:25.474318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.474301 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-openshift-ca\") pod \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " Nov 01 21:14:25.474557 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.474436 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-xks-ca\") pod \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\" (UID: \"5edda2ac-5594-45c2-b9b4-ec86240b82cc\") " Nov 01 21:14:25.474722 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.474680 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-openshift-ca" pod "5edda2ac-5594-45c2-b9b4-ec86240b82cc" (UID: "5edda2ac-5594-45c2-b9b4-ec86240b82cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:25.477464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.477431 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-xks-ca" pod "5edda2ac-5594-45c2-b9b4-ec86240b82cc" (UID: "5edda2ac-5594-45c2-b9b4-ec86240b82cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:25.478110 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.478086 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edda2ac-5594-45c2-b9b4-ec86240b82cc-kube-api-access-7kmq2" pod "5edda2ac-5594-45c2-b9b4-ec86240b82cc" (UID: "5edda2ac-5594-45c2-b9b4-ec86240b82cc"). InnerVolumeSpecName "kube-api-access-7kmq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:25.575871 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.575826 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:25.575871 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.575868 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kmq2\" (UniqueName: \"kubernetes.io/projected/5edda2ac-5594-45c2-b9b4-ec86240b82cc-kube-api-access-7kmq2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:25.576103 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:25.575883 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5edda2ac-5594-45c2-b9b4-ec86240b82cc-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:26.225387 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:26.225354 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" Nov 01 21:14:26.225583 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:26.225361 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn" event={"ID":"5edda2ac-5594-45c2-b9b4-ec86240b82cc","Type":"ContainerDied","Data":"058ade50a3cab6e30a307a458d1861dc4f59db71863212dc4cdb17b307a8d003"} Nov 01 21:14:26.225583 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:26.225482 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058ade50a3cab6e30a307a458d1861dc4f59db71863212dc4cdb17b307a8d003" Nov 01 21:14:26.370893 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:26.370855 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:31.003089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:31.003043 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:31.681480 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:31.681447 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:14:32.141931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:32.141890 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:34.758467 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:34.758425 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:34.957865 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:34.957827 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:35.527813 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.527777 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp"] Nov 01 21:14:35.528272 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.528259 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5edda2ac-5594-45c2-b9b4-ec86240b82cc" containerName="revoke-keys" Nov 01 21:14:35.528315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.528275 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edda2ac-5594-45c2-b9b4-ec86240b82cc" containerName="revoke-keys" Nov 01 21:14:35.528408 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.528397 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edda2ac-5594-45c2-b9b4-ec86240b82cc" containerName="revoke-keys" Nov 01 21:14:35.531884 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.531866 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.588972 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.587276 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp"] Nov 01 21:14:35.632873 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.632840 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:35.634436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.633503 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:14:35.634579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.633590 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:35.634579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.633636 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:35.636760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.636727 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8"] Nov 01 21:14:35.638116 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.638087 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:35.640903 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.640876 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj"] Nov 01 21:14:35.645755 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.645732 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.676128 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.676085 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:35.682620 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.682592 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.682768 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.682738 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkq4\" (UniqueName: \"kubernetes.io/projected/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-kube-api-access-zkkq4\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.682894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.682877 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.698477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.698442 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj"] Nov 01 21:14:35.784072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.783976 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkq4\" (UniqueName: \"kubernetes.io/projected/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-kube-api-access-zkkq4\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.784622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.784085 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ggm\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-kube-api-access-26ggm\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.784622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.784114 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.784622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.784203 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.784622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.784237 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.784622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.784289 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.784622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.784316 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.785050 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.785031 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.787064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.787041 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.840009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.839967 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkq4\" (UniqueName: \"kubernetes.io/projected/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-kube-api-access-zkkq4\") pod \"maas-api-revoke-sub-5c295f91-k8sgp\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.844591 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.844558 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:35.885273 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.885236 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.885516 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.885279 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.885516 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.885311 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.885516 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.885501 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ggm\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-kube-api-access-26ggm\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.886171 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.886137 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.886288 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.886255 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.888314 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.888238 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.943736 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.943706 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ggm\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-kube-api-access-26ggm\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:35.959385 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:35.959352 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:36.219870 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:36.219831 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp"] Nov 01 21:14:36.221041 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:14:36.221004 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e65d63a_813b_4c45_95b3_08f1fbf3f4c8.slice/crio-326cd1c671630d40da5c6033f34d9d555cd9595aad422918282d04b8ef3e556a WatchSource:0}: Error finding container 326cd1c671630d40da5c6033f34d9d555cd9595aad422918282d04b8ef3e556a: Status 404 returned error can't find the container with id 326cd1c671630d40da5c6033f34d9d555cd9595aad422918282d04b8ef3e556a Nov 01 21:14:36.283726 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:36.283693 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" event={"ID":"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8","Type":"ContainerStarted","Data":"326cd1c671630d40da5c6033f34d9d555cd9595aad422918282d04b8ef3e556a"} Nov 01 21:14:36.334951 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:36.334915 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj"] Nov 01 21:14:36.336102 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:14:36.336067 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02897cac_e8f4_42b8_a51a_4e09aa1b54f0.slice/crio-0a942752659a791eb1aeef29f6b73d522b435cd252cf6607a4d34c2a664b2067 WatchSource:0}: Error finding container 0a942752659a791eb1aeef29f6b73d522b435cd252cf6607a4d34c2a664b2067: Status 404 returned error can't find the container with id 0a942752659a791eb1aeef29f6b73d522b435cd252cf6607a4d34c2a664b2067 Nov 01 21:14:37.291080 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:37.291042 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" event={"ID":"02897cac-e8f4-42b8-a51a-4e09aa1b54f0","Type":"ContainerStarted","Data":"1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e"} Nov 01 21:14:37.291610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:37.291110 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" event={"ID":"02897cac-e8f4-42b8-a51a-4e09aa1b54f0","Type":"ContainerStarted","Data":"0a942752659a791eb1aeef29f6b73d522b435cd252cf6607a4d34c2a664b2067"} Nov 01 21:14:37.291610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:37.291253 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:14:37.292871 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:37.292835 5633 generic.go:356] "Generic (PLEG): container finished" podID="0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" containerID="a68f8101eac9784fd567551b6c3da0c8ab507ffe9010982d6071da8476c463b3" exitCode=0 Nov 01 21:14:37.293007 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:37.292864 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" event={"ID":"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8","Type":"ContainerDied","Data":"a68f8101eac9784fd567551b6c3da0c8ab507ffe9010982d6071da8476c463b3"} Nov 01 21:14:37.405921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:37.405824 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" podStartSLOduration=2.405809888 podStartE2EDuration="2.405809888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:37.403765741 +0000 UTC m=+4285.078025017" watchObservedRunningTime="2026-11-01 21:14:37.405809888 +0000 UTC m=+4285.080069163" Nov 01 21:14:38.441932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.441899 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:38.613300 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.613212 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-xks-ca\") pod \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " Nov 01 21:14:38.613300 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.613253 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-openshift-ca\") pod \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " Nov 01 21:14:38.613536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.613307 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-kube-api-access-zkkq4\" (UniqueName: \"kubernetes.io/projected/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-kube-api-access-zkkq4\") pod \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\" (UID: \"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8\") " Nov 01 21:14:38.613636 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.613614 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-openshift-ca" pod "0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" (UID: "0e65d63a-813b-4c45-95b3-08f1fbf3f4c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:38.615544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.615510 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-xks-ca" pod "0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" (UID: "0e65d63a-813b-4c45-95b3-08f1fbf3f4c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:38.615674 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.615607 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-kube-api-access-zkkq4" pod "0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" (UID: "0e65d63a-813b-4c45-95b3-08f1fbf3f4c8"). InnerVolumeSpecName "kube-api-access-zkkq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:38.715184 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.715143 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkq4\" (UniqueName: \"kubernetes.io/projected/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-kube-api-access-zkkq4\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:38.715184 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.715183 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:38.715449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:38.715201 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:39.306406 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:39.306373 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" Nov 01 21:14:39.306619 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:39.306368 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp" event={"ID":"0e65d63a-813b-4c45-95b3-08f1fbf3f4c8","Type":"ContainerDied","Data":"326cd1c671630d40da5c6033f34d9d555cd9595aad422918282d04b8ef3e556a"} Nov 01 21:14:39.306619 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:39.306501 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="326cd1c671630d40da5c6033f34d9d555cd9595aad422918282d04b8ef3e556a" Nov 01 21:14:40.376838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:40.376802 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:40.637587 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:40.637490 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerName="payload-processing" containerID="cri-o://6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141" gracePeriod=25 Nov 01 21:14:40.895752 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:40.895686 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:41.038695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.038662 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-plugins-config-volume\") pod \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " Nov 01 21:14:41.038875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.038738 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-provider-credentials\") pod \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " Nov 01 21:14:41.038875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.038835 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-routing-overlay-volume\") pod \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " Nov 01 21:14:41.038875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.038867 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-kube-api-access-vs99g\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-kube-api-access-vs99g\") pod \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\" (UID: \"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4\") " Nov 01 21:14:41.039226 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.039186 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-plugins-config-volume" pod "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" (UID: "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:41.039512 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.039224 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-routing-overlay-volume" pod "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" (UID: "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:41.041674 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.041645 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-kube-api-access-vs99g" pod "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" (UID: "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4"). InnerVolumeSpecName "kube-api-access-vs99g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:41.042042 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.042018 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-provider-credentials" pod "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" (UID: "d50b6ce8-d8c7-4a93-bbc2-4791af69efb4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:41.139792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.139759 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:41.139792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.139789 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:41.139792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.139800 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs99g\" (UniqueName: \"kubernetes.io/projected/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-kube-api-access-vs99g\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:41.140045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.139811 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:41.322708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.322662 5633 generic.go:356] "Generic (PLEG): container finished" podID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerID="6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141" exitCode=0 Nov 01 21:14:41.322708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.322710 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" Nov 01 21:14:41.322926 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.322732 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" event={"ID":"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4","Type":"ContainerDied","Data":"6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141"} Nov 01 21:14:41.322926 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.322803 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8" event={"ID":"d50b6ce8-d8c7-4a93-bbc2-4791af69efb4","Type":"ContainerDied","Data":"6500103a60ddb20cf3e06a74a70fa31572fde106928ee91e17d041c5c664a278"} Nov 01 21:14:41.322926 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.322834 5633 scope.go:122] "RemoveContainer" containerID="6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141" Nov 01 21:14:41.333118 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.333096 5633 scope.go:122] "RemoveContainer" containerID="6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141" Nov 01 21:14:41.333454 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:41.333428 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141\": container with ID starting with 6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141 not found: ID does not exist" containerID="6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141" Nov 01 21:14:41.333532 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.333463 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141"} err="failed to get container status \"6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141\": rpc error: code = NotFound desc = could not find container \"6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141\": container with ID starting with 6f7ad048f848b63fa23b1655ad305e25d41ecc2aae2aaece8f62c03f87090141 not found: ID does not exist" Nov 01 21:14:41.485015 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.484978 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8"] Nov 01 21:14:41.534552 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.534525 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cdm8zn8"] Nov 01 21:14:41.853063 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:41.853025 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:42.467808 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:42.467774 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" path="/var/lib/kubelet/pods/d50b6ce8-d8c7-4a93-bbc2-4791af69efb4/volumes" Nov 01 21:14:45.083367 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:45.083298 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:45.511659 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:45.511619 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:14:46.156887 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:14:46.156843 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:14:47.206085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.206042 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t"] Nov 01 21:14:47.206765 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.206743 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" containerName="revoke-keys" Nov 01 21:14:47.206868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.206767 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" containerName="revoke-keys" Nov 01 21:14:47.206868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.206836 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerName="payload-processing" Nov 01 21:14:47.206868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.206846 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerName="payload-processing" Nov 01 21:14:47.207043 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.206961 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50b6ce8-d8c7-4a93-bbc2-4791af69efb4" containerName="payload-processing" Nov 01 21:14:47.207043 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.206977 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" containerName="revoke-keys" Nov 01 21:14:47.211786 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.211763 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.266267 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.266231 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t"] Nov 01 21:14:47.298631 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.298589 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44qqr\" (UniqueName: \"kubernetes.io/projected/638a104a-f1f9-41ac-891e-9ccad23d2540-kube-api-access-44qqr\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.298631 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.298635 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.298857 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.298739 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.311704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.311671 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:14:47.311704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.311691 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:14:47.311704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.311706 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:14:47.313080 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.313059 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:14:47.400243 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.400210 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44qqr\" (UniqueName: \"kubernetes.io/projected/638a104a-f1f9-41ac-891e-9ccad23d2540-kube-api-access-44qqr\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.400243 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.400253 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.400550 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.400363 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.401019 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.400975 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.402932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.402912 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.456398 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.456286 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44qqr\" (UniqueName: \"kubernetes.io/projected/638a104a-f1f9-41ac-891e-9ccad23d2540-kube-api-access-44qqr\") pod \"maas-api-revoke-sub-e10c03ca-pxc6t\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.527042 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.527004 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:47.904899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:47.904859 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t"] Nov 01 21:14:47.905449 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:14:47.905422 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638a104a_f1f9_41ac_891e_9ccad23d2540.slice/crio-eae7d113aef2bb59c962f1e2cdfc6adea99656592e7126a3def9b3bd521eaf86 WatchSource:0}: Error finding container eae7d113aef2bb59c962f1e2cdfc6adea99656592e7126a3def9b3bd521eaf86: Status 404 returned error can't find the container with id eae7d113aef2bb59c962f1e2cdfc6adea99656592e7126a3def9b3bd521eaf86 Nov 01 21:14:48.302608 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:48.302571 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:48.365953 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:48.365908 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" event={"ID":"638a104a-f1f9-41ac-891e-9ccad23d2540","Type":"ContainerStarted","Data":"7989d3e723a03cca7e5eab1c48343a7f4ad3b40760e61030eb45d4a7415ff698"} Nov 01 21:14:48.366146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:48.365967 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" event={"ID":"638a104a-f1f9-41ac-891e-9ccad23d2540","Type":"ContainerStarted","Data":"eae7d113aef2bb59c962f1e2cdfc6adea99656592e7126a3def9b3bd521eaf86"} Nov 01 21:14:48.479674 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:48.479617 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" podStartSLOduration=1.479600906 podStartE2EDuration="1.479600906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:48.476516716 +0000 UTC m=+4296.150775972" watchObservedRunningTime="2026-11-01 21:14:48.479600906 +0000 UTC m=+4296.153860192" Nov 01 21:14:49.373067 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:49.373028 5633 generic.go:356] "Generic (PLEG): container finished" podID="638a104a-f1f9-41ac-891e-9ccad23d2540" containerID="7989d3e723a03cca7e5eab1c48343a7f4ad3b40760e61030eb45d4a7415ff698" exitCode=0 Nov 01 21:14:49.373495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:49.373114 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" event={"ID":"638a104a-f1f9-41ac-891e-9ccad23d2540","Type":"ContainerDied","Data":"7989d3e723a03cca7e5eab1c48343a7f4ad3b40760e61030eb45d4a7415ff698"} Nov 01 21:14:49.783549 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:49.783507 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:50.525083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.525010 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:50.640962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.640872 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-openshift-ca\") pod \"638a104a-f1f9-41ac-891e-9ccad23d2540\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " Nov 01 21:14:50.640962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.640946 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-xks-ca\") pod \"638a104a-f1f9-41ac-891e-9ccad23d2540\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " Nov 01 21:14:50.641212 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.640987 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638a104a-f1f9-41ac-891e-9ccad23d2540-kube-api-access-44qqr\" (UniqueName: \"kubernetes.io/projected/638a104a-f1f9-41ac-891e-9ccad23d2540-kube-api-access-44qqr\") pod \"638a104a-f1f9-41ac-891e-9ccad23d2540\" (UID: \"638a104a-f1f9-41ac-891e-9ccad23d2540\") " Nov 01 21:14:50.641352 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.641306 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-openshift-ca" pod "638a104a-f1f9-41ac-891e-9ccad23d2540" (UID: "638a104a-f1f9-41ac-891e-9ccad23d2540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:50.643413 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.643381 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638a104a-f1f9-41ac-891e-9ccad23d2540-kube-api-access-44qqr" pod "638a104a-f1f9-41ac-891e-9ccad23d2540" (UID: "638a104a-f1f9-41ac-891e-9ccad23d2540"). InnerVolumeSpecName "kube-api-access-44qqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:50.643665 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.643637 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-xks-ca" pod "638a104a-f1f9-41ac-891e-9ccad23d2540" (UID: "638a104a-f1f9-41ac-891e-9ccad23d2540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:14:50.742039 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.741998 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:50.742039 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.742033 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638a104a-f1f9-41ac-891e-9ccad23d2540-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:50.742039 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:50.742043 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44qqr\" (UniqueName: \"kubernetes.io/projected/638a104a-f1f9-41ac-891e-9ccad23d2540-kube-api-access-44qqr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:51.386676 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:51.386643 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" Nov 01 21:14:51.386897 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:51.386684 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t" event={"ID":"638a104a-f1f9-41ac-891e-9ccad23d2540","Type":"ContainerDied","Data":"eae7d113aef2bb59c962f1e2cdfc6adea99656592e7126a3def9b3bd521eaf86"} Nov 01 21:14:51.386897 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:51.386734 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae7d113aef2bb59c962f1e2cdfc6adea99656592e7126a3def9b3bd521eaf86" Nov 01 21:14:54.481575 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:54.481538 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:57.299257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:57.299214 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:14:58.300948 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:58.300904 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:59.269710 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:59.269672 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:14:59.947559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:59.947529 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj"] Nov 01 21:14:59.948923 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:14:59.948892 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:15:00.077166 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.077127 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:00.168148 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.168112 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78"] Nov 01 21:15:00.168641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.168625 5633 cpu_manager.go:414] "containerMap: removing container" podUID="638a104a-f1f9-41ac-891e-9ccad23d2540" containerName="revoke-keys" Nov 01 21:15:00.168700 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.168643 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638a104a-f1f9-41ac-891e-9ccad23d2540" containerName="revoke-keys" Nov 01 21:15:00.168737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.168731 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="638a104a-f1f9-41ac-891e-9ccad23d2540" containerName="revoke-keys" Nov 01 21:15:00.172174 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.172156 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw"] Nov 01 21:15:00.172291 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.172270 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.175812 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.175791 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.226204 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.226163 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:15:00.226204 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.226187 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:15:00.226510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.226270 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:15:00.229063 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.229017 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78"] Nov 01 21:15:00.232614 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.232590 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw"] Nov 01 21:15:00.242905 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.242869 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35103c1f-a5da-47b0-a0fc-8e834459cd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.243062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.242939 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbv9j\" (UniqueName: \"kubernetes.io/projected/35103c1f-a5da-47b0-a0fc-8e834459cd06-kube-api-access-lbv9j\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.243062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.242999 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.243137 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.243083 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgws\" (UniqueName: \"kubernetes.io/projected/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-kube-api-access-bqgws\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.252196 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.252162 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:00.344575 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.344535 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35103c1f-a5da-47b0-a0fc-8e834459cd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.344791 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.344600 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbv9j\" (UniqueName: \"kubernetes.io/projected/35103c1f-a5da-47b0-a0fc-8e834459cd06-kube-api-access-lbv9j\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.344791 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.344635 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.344791 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.344671 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgws\" (UniqueName: \"kubernetes.io/projected/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-kube-api-access-bqgws\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.345299 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.345281 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35103c1f-a5da-47b0-a0fc-8e834459cd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.345363 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.345287 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.401101 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.401064 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbv9j\" (UniqueName: \"kubernetes.io/projected/35103c1f-a5da-47b0-a0fc-8e834459cd06-kube-api-access-lbv9j\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.401264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.401064 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgws\" (UniqueName: \"kubernetes.io/projected/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-kube-api-access-bqgws\") pod \"maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.487613 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.487536 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:00.493789 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.493767 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:00.771445 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.771350 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:00.863590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.863549 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78"] Nov 01 21:15:00.904721 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:15:00.904678 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86414fcd_5b69_4886_9a0a_4fa0c733b9f3.slice/crio-1548a0fc74757cc97858a031d9d5091c10ff8acf4a83ca64aed3728a1ecc6dbd WatchSource:0}: Error finding container 1548a0fc74757cc97858a031d9d5091c10ff8acf4a83ca64aed3728a1ecc6dbd: Status 404 returned error can't find the container with id 1548a0fc74757cc97858a031d9d5091c10ff8acf4a83ca64aed3728a1ecc6dbd Nov 01 21:15:00.913154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:00.913124 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw"] Nov 01 21:15:00.916505 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:15:00.916464 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35103c1f_a5da_47b0_a0fc_8e834459cd06.slice/crio-1731110ce3558a5715cd190a2000bd1cec48f09c5d1990f7d1b28832cca5ace8 WatchSource:0}: Error finding container 1731110ce3558a5715cd190a2000bd1cec48f09c5d1990f7d1b28832cca5ace8: Status 404 returned error can't find the container with id 1731110ce3558a5715cd190a2000bd1cec48f09c5d1990f7d1b28832cca5ace8 Nov 01 21:15:01.447630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:01.447522 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" event={"ID":"35103c1f-a5da-47b0-a0fc-8e834459cd06","Type":"ContainerStarted","Data":"cc1b4c9dc3038b99f0a86ee9f57b0dd951900ac63b206b951475140912ff8310"} Nov 01 21:15:01.447630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:01.447579 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" event={"ID":"35103c1f-a5da-47b0-a0fc-8e834459cd06","Type":"ContainerStarted","Data":"1731110ce3558a5715cd190a2000bd1cec48f09c5d1990f7d1b28832cca5ace8"} Nov 01 21:15:01.449214 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:01.449188 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" event={"ID":"86414fcd-5b69-4886-9a0a-4fa0c733b9f3","Type":"ContainerStarted","Data":"f132f7451bf69251f34deec29b21890bee2a4a81547f592ec3f6a664d7126b42"} Nov 01 21:15:01.449379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:01.449236 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" event={"ID":"86414fcd-5b69-4886-9a0a-4fa0c733b9f3","Type":"ContainerStarted","Data":"1548a0fc74757cc97858a031d9d5091c10ff8acf4a83ca64aed3728a1ecc6dbd"} Nov 01 21:15:01.559892 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:01.559839 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" podStartSLOduration=1.559824262 podStartE2EDuration="1.559824262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:01.558402766 +0000 UTC m=+4309.232662022" watchObservedRunningTime="2026-11-01 21:15:01.559824262 +0000 UTC m=+4309.234083516" Nov 01 21:15:01.669818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:01.669748 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" podStartSLOduration=1.669728438 podStartE2EDuration="1.669728438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:01.668374756 +0000 UTC m=+4309.342634013" watchObservedRunningTime="2026-11-01 21:15:01.669728438 +0000 UTC m=+4309.343987693" Nov 01 21:15:02.457233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:02.457195 5633 generic.go:356] "Generic (PLEG): container finished" podID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerID="f132f7451bf69251f34deec29b21890bee2a4a81547f592ec3f6a664d7126b42" exitCode=60 Nov 01 21:15:02.457709 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:02.457289 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" event={"ID":"86414fcd-5b69-4886-9a0a-4fa0c733b9f3","Type":"ContainerDied","Data":"f132f7451bf69251f34deec29b21890bee2a4a81547f592ec3f6a664d7126b42"} Nov 01 21:15:02.457709 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:02.457539 5633 scope.go:122] "RemoveContainer" containerID="f132f7451bf69251f34deec29b21890bee2a4a81547f592ec3f6a664d7126b42" Nov 01 21:15:02.459117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:02.459093 5633 generic.go:356] "Generic (PLEG): container finished" podID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerID="cc1b4c9dc3038b99f0a86ee9f57b0dd951900ac63b206b951475140912ff8310" exitCode=60 Nov 01 21:15:02.459427 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:02.459412 5633 scope.go:122] "RemoveContainer" containerID="cc1b4c9dc3038b99f0a86ee9f57b0dd951900ac63b206b951475140912ff8310" Nov 01 21:15:02.466500 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:02.466469 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" event={"ID":"35103c1f-a5da-47b0-a0fc-8e834459cd06","Type":"ContainerDied","Data":"cc1b4c9dc3038b99f0a86ee9f57b0dd951900ac63b206b951475140912ff8310"} Nov 01 21:15:02.753688 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:02.753648 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:03.467459 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.467426 5633 generic.go:356] "Generic (PLEG): container finished" podID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerID="cf2cbe03a7338a7f3188dd15d1d31a73882ba6c748957b966de0a9ab377c5ccd" exitCode=60 Nov 01 21:15:03.467931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.467498 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" event={"ID":"35103c1f-a5da-47b0-a0fc-8e834459cd06","Type":"ContainerDied","Data":"cf2cbe03a7338a7f3188dd15d1d31a73882ba6c748957b966de0a9ab377c5ccd"} Nov 01 21:15:03.467931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.467551 5633 scope.go:122] "RemoveContainer" containerID="cc1b4c9dc3038b99f0a86ee9f57b0dd951900ac63b206b951475140912ff8310" Nov 01 21:15:03.467931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.467860 5633 scope.go:122] "RemoveContainer" containerID="cf2cbe03a7338a7f3188dd15d1d31a73882ba6c748957b966de0a9ab377c5ccd" Nov 01 21:15:03.468168 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:03.468143 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw_odh-ai-gateway-infra(35103c1f-a5da-47b0-a0fc-8e834459cd06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" Nov 01 21:15:03.469828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.469806 5633 generic.go:356] "Generic (PLEG): container finished" podID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerID="e623135994f3fdda281ac09c18fe4613876e41369683a8d5637f3d09c047bf01" exitCode=60 Nov 01 21:15:03.469929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.469846 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" event={"ID":"86414fcd-5b69-4886-9a0a-4fa0c733b9f3","Type":"ContainerDied","Data":"e623135994f3fdda281ac09c18fe4613876e41369683a8d5637f3d09c047bf01"} Nov 01 21:15:03.470122 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.470110 5633 scope.go:122] "RemoveContainer" containerID="e623135994f3fdda281ac09c18fe4613876e41369683a8d5637f3d09c047bf01" Nov 01 21:15:03.470370 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:03.470347 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78_odh-ai-gateway-infra(86414fcd-5b69-4886-9a0a-4fa0c733b9f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" Nov 01 21:15:03.480953 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:03.480931 5633 scope.go:122] "RemoveContainer" containerID="f132f7451bf69251f34deec29b21890bee2a4a81547f592ec3f6a664d7126b42" Nov 01 21:15:04.016380 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:04.016324 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:04.480795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:04.480768 5633 scope.go:122] "RemoveContainer" containerID="cf2cbe03a7338a7f3188dd15d1d31a73882ba6c748957b966de0a9ab377c5ccd" Nov 01 21:15:04.481258 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:04.481086 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw_odh-ai-gateway-infra(35103c1f-a5da-47b0-a0fc-8e834459cd06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" Nov 01 21:15:04.482279 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:04.482263 5633 scope.go:122] "RemoveContainer" containerID="e623135994f3fdda281ac09c18fe4613876e41369683a8d5637f3d09c047bf01" Nov 01 21:15:04.482530 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:04.482511 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78_odh-ai-gateway-infra(86414fcd-5b69-4886-9a0a-4fa0c733b9f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" Nov 01 21:15:04.948256 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:04.948156 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerName="payload-processing" containerID="cri-o://1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e" gracePeriod=25 Nov 01 21:15:05.214621 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.214593 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:15:05.294685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.294653 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-plugins-config-volume\") pod \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " Nov 01 21:15:05.294878 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.294747 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-provider-credentials\") pod \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " Nov 01 21:15:05.294878 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.294819 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-kube-api-access-26ggm\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-kube-api-access-26ggm\") pod \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " Nov 01 21:15:05.294878 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.294872 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-routing-overlay-volume\") pod \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\" (UID: \"02897cac-e8f4-42b8-a51a-4e09aa1b54f0\") " Nov 01 21:15:05.295050 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.295030 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-plugins-config-volume" pod "02897cac-e8f4-42b8-a51a-4e09aa1b54f0" (UID: "02897cac-e8f4-42b8-a51a-4e09aa1b54f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:05.295207 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.295186 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:05.295302 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.295255 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-routing-overlay-volume" pod "02897cac-e8f4-42b8-a51a-4e09aa1b54f0" (UID: "02897cac-e8f4-42b8-a51a-4e09aa1b54f0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:05.297163 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.297138 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-provider-credentials" pod "02897cac-e8f4-42b8-a51a-4e09aa1b54f0" (UID: "02897cac-e8f4-42b8-a51a-4e09aa1b54f0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:05.297163 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.297151 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-kube-api-access-26ggm" pod "02897cac-e8f4-42b8-a51a-4e09aa1b54f0" (UID: "02897cac-e8f4-42b8-a51a-4e09aa1b54f0"). InnerVolumeSpecName "kube-api-access-26ggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:05.395962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.395918 5633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-provider-credentials\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:05.395962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.395953 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26ggm\" (UniqueName: \"kubernetes.io/projected/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-kube-api-access-26ggm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:05.395962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.395963 5633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/02897cac-e8f4-42b8-a51a-4e09aa1b54f0-routing-overlay-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:05.489637 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.489541 5633 generic.go:356] "Generic (PLEG): container finished" podID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerID="1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e" exitCode=0 Nov 01 21:15:05.489637 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.489589 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" Nov 01 21:15:05.489637 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.489618 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" event={"ID":"02897cac-e8f4-42b8-a51a-4e09aa1b54f0","Type":"ContainerDied","Data":"1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e"} Nov 01 21:15:05.490183 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.489672 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj" event={"ID":"02897cac-e8f4-42b8-a51a-4e09aa1b54f0","Type":"ContainerDied","Data":"0a942752659a791eb1aeef29f6b73d522b435cd252cf6607a4d34c2a664b2067"} Nov 01 21:15:05.490183 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.489702 5633 scope.go:122] "RemoveContainer" containerID="1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e" Nov 01 21:15:05.500026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.500002 5633 scope.go:122] "RemoveContainer" containerID="1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e" Nov 01 21:15:05.500360 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:05.500313 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e\": container with ID starting with 1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e not found: ID does not exist" containerID="1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e" Nov 01 21:15:05.500431 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.500362 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e"} err="failed to get container status \"1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e\": rpc error: code = NotFound desc = could not find container \"1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e\": container with ID starting with 1ea42b8355abcd3ec9dedb7afdece71a16fd46d268cdaff2b46e3b45ea66900e not found: ID does not exist" Nov 01 21:15:05.652414 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.652349 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj"] Nov 01 21:15:05.655901 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:05.655875 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddcfgsj"] Nov 01 21:15:06.466285 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:06.466248 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" path="/var/lib/kubelet/pods/02897cac-e8f4-42b8-a51a-4e09aa1b54f0/volumes" Nov 01 21:15:06.564305 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:06.564259 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:06.683970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:06.683928 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:06.983765 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:06.983731 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:08.694432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:08.694397 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:11.611458 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:11.611418 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:15:12.619718 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:12.619682 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:12.792115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:12.792071 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:13.350940 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:13.350905 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:13.524550 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:13.524519 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:15.460118 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:15.460089 5633 scope.go:122] "RemoveContainer" containerID="cf2cbe03a7338a7f3188dd15d1d31a73882ba6c748957b966de0a9ab377c5ccd" Nov 01 21:15:16.157364 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:16.157256 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:15:16.460180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:16.460101 5633 scope.go:122] "RemoveContainer" containerID="e623135994f3fdda281ac09c18fe4613876e41369683a8d5637f3d09c047bf01" Nov 01 21:15:16.557712 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:16.557680 5633 generic.go:356] "Generic (PLEG): container finished" podID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerID="26c04e30c719e073c77cd5ea74d5b7452329f739ecafc90b8b3d6d4ac12c5308" exitCode=60 Nov 01 21:15:16.557888 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:16.557757 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" event={"ID":"35103c1f-a5da-47b0-a0fc-8e834459cd06","Type":"ContainerDied","Data":"26c04e30c719e073c77cd5ea74d5b7452329f739ecafc90b8b3d6d4ac12c5308"} Nov 01 21:15:16.557888 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:16.557817 5633 scope.go:122] "RemoveContainer" containerID="cf2cbe03a7338a7f3188dd15d1d31a73882ba6c748957b966de0a9ab377c5ccd" Nov 01 21:15:16.558181 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:16.558166 5633 scope.go:122] "RemoveContainer" containerID="26c04e30c719e073c77cd5ea74d5b7452329f739ecafc90b8b3d6d4ac12c5308" Nov 01 21:15:16.558499 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:16.558480 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw_odh-ai-gateway-infra(35103c1f-a5da-47b0-a0fc-8e834459cd06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" Nov 01 21:15:17.565436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.565324 5633 generic.go:356] "Generic (PLEG): container finished" podID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerID="5ebed7f89dbd4de7dc19b279127a5afd7596c5301029d8b0899e4d63c092e1ae" exitCode=60 Nov 01 21:15:17.565436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.565372 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" event={"ID":"86414fcd-5b69-4886-9a0a-4fa0c733b9f3","Type":"ContainerDied","Data":"5ebed7f89dbd4de7dc19b279127a5afd7596c5301029d8b0899e4d63c092e1ae"} Nov 01 21:15:17.565970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.565439 5633 scope.go:122] "RemoveContainer" containerID="e623135994f3fdda281ac09c18fe4613876e41369683a8d5637f3d09c047bf01" Nov 01 21:15:17.565970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.565782 5633 scope.go:122] "RemoveContainer" containerID="5ebed7f89dbd4de7dc19b279127a5afd7596c5301029d8b0899e4d63c092e1ae" Nov 01 21:15:17.566087 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:17.566044 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78_odh-ai-gateway-infra(86414fcd-5b69-4886-9a0a-4fa0c733b9f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" Nov 01 21:15:17.679917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.679884 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw"] Nov 01 21:15:17.812375 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.812275 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:17.831131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.831069 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35103c1f-a5da-47b0-a0fc-8e834459cd06-kube-api-access-lbv9j\" (UniqueName: \"kubernetes.io/projected/35103c1f-a5da-47b0-a0fc-8e834459cd06-kube-api-access-lbv9j\") pod \"35103c1f-a5da-47b0-a0fc-8e834459cd06\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " Nov 01 21:15:17.833322 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.833295 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35103c1f-a5da-47b0-a0fc-8e834459cd06-kube-api-access-lbv9j" pod "35103c1f-a5da-47b0-a0fc-8e834459cd06" (UID: "35103c1f-a5da-47b0-a0fc-8e834459cd06"). InnerVolumeSpecName "kube-api-access-lbv9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:17.931762 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.931724 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35103c1f-a5da-47b0-a0fc-8e834459cd06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35103c1f-a5da-47b0-a0fc-8e834459cd06-maas-api-service-ca\") pod \"35103c1f-a5da-47b0-a0fc-8e834459cd06\" (UID: \"35103c1f-a5da-47b0-a0fc-8e834459cd06\") " Nov 01 21:15:17.932117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.932090 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbv9j\" (UniqueName: \"kubernetes.io/projected/35103c1f-a5da-47b0-a0fc-8e834459cd06-kube-api-access-lbv9j\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:17.932261 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:17.932117 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35103c1f-a5da-47b0-a0fc-8e834459cd06-maas-api-service-ca" pod "35103c1f-a5da-47b0-a0fc-8e834459cd06" (UID: "35103c1f-a5da-47b0-a0fc-8e834459cd06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:18.033360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.033312 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35103c1f-a5da-47b0-a0fc-8e834459cd06-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:18.459921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.459886 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:18.576930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.576893 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" event={"ID":"35103c1f-a5da-47b0-a0fc-8e834459cd06","Type":"ContainerDied","Data":"1731110ce3558a5715cd190a2000bd1cec48f09c5d1990f7d1b28832cca5ace8"} Nov 01 21:15:18.576930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.576935 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw" Nov 01 21:15:18.577442 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.576965 5633 scope.go:122] "RemoveContainer" containerID="26c04e30c719e073c77cd5ea74d5b7452329f739ecafc90b8b3d6d4ac12c5308" Nov 01 21:15:18.690558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.690471 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78"] Nov 01 21:15:18.738280 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.738243 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw"] Nov 01 21:15:18.742037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.742010 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892795-ccflw"] Nov 01 21:15:18.830379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.830351 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:18.840089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.840062 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-maas-api-service-ca\") pod \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " Nov 01 21:15:18.840242 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.840113 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-kube-api-access-bqgws\" (UniqueName: \"kubernetes.io/projected/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-kube-api-access-bqgws\") pod \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\" (UID: \"86414fcd-5b69-4886-9a0a-4fa0c733b9f3\") " Nov 01 21:15:18.840521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.840493 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-maas-api-service-ca" pod "86414fcd-5b69-4886-9a0a-4fa0c733b9f3" (UID: "86414fcd-5b69-4886-9a0a-4fa0c733b9f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:18.842371 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.842350 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-kube-api-access-bqgws" pod "86414fcd-5b69-4886-9a0a-4fa0c733b9f3" (UID: "86414fcd-5b69-4886-9a0a-4fa0c733b9f3"). InnerVolumeSpecName "kube-api-access-bqgws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:18.941704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.941610 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:18.941704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:18.941642 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgws\" (UniqueName: \"kubernetes.io/projected/86414fcd-5b69-4886-9a0a-4fa0c733b9f3-kube-api-access-bqgws\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:19.188226 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:19.188184 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:19.583325 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:19.583293 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" Nov 01 21:15:19.583838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:19.583354 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78" event={"ID":"86414fcd-5b69-4886-9a0a-4fa0c733b9f3","Type":"ContainerDied","Data":"1548a0fc74757cc97858a031d9d5091c10ff8acf4a83ca64aed3728a1ecc6dbd"} Nov 01 21:15:19.583838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:19.583423 5633 scope.go:122] "RemoveContainer" containerID="5ebed7f89dbd4de7dc19b279127a5afd7596c5301029d8b0899e4d63c092e1ae" Nov 01 21:15:19.746592 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:19.746557 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78"] Nov 01 21:15:19.795481 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:19.795434 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6a675a-29892795-wxv78"] Nov 01 21:15:20.466188 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:20.466158 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" path="/var/lib/kubelet/pods/35103c1f-a5da-47b0-a0fc-8e834459cd06/volumes" Nov 01 21:15:20.466599 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:20.466585 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" path="/var/lib/kubelet/pods/86414fcd-5b69-4886-9a0a-4fa0c733b9f3/volumes" Nov 01 21:15:22.583010 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:22.582931 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:24.301621 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:24.301580 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:24.618137 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:24.618049 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:24.747685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:24.747651 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:26.084178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:26.084133 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:26.690084 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:26.690047 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:26.704188 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:26.704160 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b"] Nov 01 21:15:26.704424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:26.704389 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" containerID="cri-o://11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a" gracePeriod=30 Nov 01 21:15:27.378526 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:27.378492 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:28.893041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:28.893004 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:15:29.284175 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:29.284133 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:31.962324 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:31.962290 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 21:15:32.077311 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077221 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-credential-socket\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077311 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077263 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-data\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077311 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077285 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-podinfo\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077311 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077312 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ede63f-d465-4d69-bda7-3800a6f7d614-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ede63f-d465-4d69-bda7-3800a6f7d614-istiod-ca-cert\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077379 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-token\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-token\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077439 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-certs\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077464 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-kube-api-access-gps49\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-kube-api-access-gps49\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077573 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-envoy\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.077697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077585 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-credential-socket" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:32.077697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077606 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-data" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:32.077697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077649 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-socket\") pod \"61ede63f-d465-4d69-bda7-3800a6f7d614\" (UID: \"61ede63f-d465-4d69-bda7-3800a6f7d614\") " Nov 01 21:15:32.078069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077768 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-certs" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:32.078069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.077887 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ede63f-d465-4d69-bda7-3800a6f7d614-istiod-ca-cert" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:32.078069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.078029 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.078069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.078049 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ede63f-d465-4d69-bda7-3800a6f7d614-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.078069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.078063 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.078279 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.078078 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.078279 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.078032 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-socket" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:32.080471 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.080449 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-podinfo" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:15:32.080471 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.080461 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-envoy" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:15:32.080587 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.080487 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-token" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:32.080587 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.080495 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-kube-api-access-gps49" pod "61ede63f-d465-4d69-bda7-3800a6f7d614" (UID: "61ede63f-d465-4d69-bda7-3800a6f7d614"). InnerVolumeSpecName "kube-api-access-gps49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:32.179155 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.179116 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.179155 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.179151 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.179155 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.179161 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gps49\" (UniqueName: \"kubernetes.io/projected/61ede63f-d465-4d69-bda7-3800a6f7d614-kube-api-access-gps49\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.179452 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.179172 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.179452 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.179182 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ede63f-d465-4d69-bda7-3800a6f7d614-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:32.660111 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.660074 5633 generic.go:356] "Generic (PLEG): container finished" podID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerID="11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a" exitCode=0 Nov 01 21:15:32.660111 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.660122 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" event={"ID":"61ede63f-d465-4d69-bda7-3800a6f7d614","Type":"ContainerDied","Data":"11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a"} Nov 01 21:15:32.660432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.660132 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" Nov 01 21:15:32.660432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.660176 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b" event={"ID":"61ede63f-d465-4d69-bda7-3800a6f7d614","Type":"ContainerDied","Data":"e6c490d62d4ea87dce6748435e85e5aeea33d64c56b5a9ed05436c091abe0625"} Nov 01 21:15:32.660432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.660208 5633 scope.go:122] "RemoveContainer" containerID="11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a" Nov 01 21:15:32.670654 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.670629 5633 scope.go:122] "RemoveContainer" containerID="49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6" Nov 01 21:15:32.682357 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.682315 5633 scope.go:122] "RemoveContainer" containerID="11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a" Nov 01 21:15:32.682720 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:32.682690 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a\": container with ID starting with 11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a not found: ID does not exist" containerID="11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a" Nov 01 21:15:32.682808 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.682733 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a"} err="failed to get container status \"11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a\": rpc error: code = NotFound desc = could not find container \"11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a\": container with ID starting with 11531fa797b4e52aaa932bc39f3f23e1cb8ee12fdb256ac6bb3de40aa7c49d6a not found: ID does not exist" Nov 01 21:15:32.682808 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.682758 5633 scope.go:122] "RemoveContainer" containerID="49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6" Nov 01 21:15:32.683065 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:32.683044 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6\": container with ID starting with 49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6 not found: ID does not exist" containerID="49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6" Nov 01 21:15:32.683127 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.683074 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6"} err="failed to get container status \"49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6\": rpc error: code = NotFound desc = could not find container \"49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6\": container with ID starting with 49aca351861aed843e70a622d3a7ca628acd37efb3ec1e50b74d0c87b72301b6 not found: ID does not exist" Nov 01 21:15:32.824517 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.824432 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b"] Nov 01 21:15:32.874052 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:32.874011 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-99d504-openshift-default-8547bd77df-b2q9b"] Nov 01 21:15:33.532281 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:33.532241 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:34.116224 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:34.116189 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:34.465970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:34.465940 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" path="/var/lib/kubelet/pods/61ede63f-d465-4d69-bda7-3800a6f7d614/volumes" Nov 01 21:15:34.753050 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:34.752958 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph"] Nov 01 21:15:34.994536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:34.994505 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:35.119228 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:35.119144 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:38.990256 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:38.990212 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:39.110284 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:39.110220 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:39.753477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:39.753407 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" podUID="20986a23-c21b-4951-98be-b22b660f14b2" containerName="payload-pre-processing" containerID="cri-o://21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022" gracePeriod=25 Nov 01 21:15:42.194770 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:42.194732 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:43.103591 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:43.103548 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:44.445985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:44.445947 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:44.620667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:44.620623 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:45.942666 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:45.942630 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:46.155714 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:15:46.155678 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:15:46.421641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:46.421604 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:15:47.270322 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:47.270285 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cf654df49-4ch47_f1ef9e76-0e8b-4050-b269-5042540f7826/authorino/0.log" Nov 01 21:15:47.415600 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:47.415557 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" podUID="20986a23-c21b-4951-98be-b22b660f14b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:15:48.159030 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:48.159002 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/2.log" Nov 01 21:15:49.473157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:49.473118 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:15:50.045274 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:50.045246 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ld4qw_f956c89a-217e-4237-a2cb-70632240750d/limitador/0.log" Nov 01 21:15:50.265749 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:50.265710 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wdb8s_a8e46f26-efa8-4823-b684-b805cce7bf88/manager/0.log" Nov 01 21:15:51.023297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:51.023260 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/1.log" Nov 01 21:15:51.560230 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:51.560202 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/1.log" Nov 01 21:15:51.913872 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:51.913782 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/1.log" Nov 01 21:15:52.822322 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:52.822286 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645c58c6f7-mc6zb_f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864/manager/1.log" Nov 01 21:15:53.248778 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:53.248750 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nscg4_775c372c-52f9-4571-b7c1-28905e6f854c/server/0.log" Nov 01 21:15:53.470907 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:53.470870 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wpr7v_d5778aa7-6472-4e46-afa1-4048cb03ef96/manager/0.log" Nov 01 21:15:53.725760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:53.725733 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjv47_292d7509-aef9-452f-ab7f-85e1117d9652/proxy/0.log" Nov 01 21:15:54.342671 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:54.342635 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/0.log" Nov 01 21:15:54.559167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:54.559137 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/0.log" Nov 01 21:15:54.889347 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:54.889310 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645c58c6f7-mc6zb_f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864/manager/0.log" Nov 01 21:15:57.415616 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:15:57.415568 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" podUID="20986a23-c21b-4951-98be-b22b660f14b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:16:00.003255 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.003231 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 21:16:00.051528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.051442 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20986a23-c21b-4951-98be-b22b660f14b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20986a23-c21b-4951-98be-b22b660f14b2-plugins-config-volume\") pod \"20986a23-c21b-4951-98be-b22b660f14b2\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " Nov 01 21:16:00.051689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.051559 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20986a23-c21b-4951-98be-b22b660f14b2-kube-api-access-4tq6w\" (UniqueName: \"kubernetes.io/projected/20986a23-c21b-4951-98be-b22b660f14b2-kube-api-access-4tq6w\") pod \"20986a23-c21b-4951-98be-b22b660f14b2\" (UID: \"20986a23-c21b-4951-98be-b22b660f14b2\") " Nov 01 21:16:00.051815 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.051791 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20986a23-c21b-4951-98be-b22b660f14b2-plugins-config-volume" pod "20986a23-c21b-4951-98be-b22b660f14b2" (UID: "20986a23-c21b-4951-98be-b22b660f14b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:00.053885 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.053865 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20986a23-c21b-4951-98be-b22b660f14b2-kube-api-access-4tq6w" pod "20986a23-c21b-4951-98be-b22b660f14b2" (UID: "20986a23-c21b-4951-98be-b22b660f14b2"). InnerVolumeSpecName "kube-api-access-4tq6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:00.152269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.152233 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20986a23-c21b-4951-98be-b22b660f14b2-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:00.152269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.152266 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tq6w\" (UniqueName: \"kubernetes.io/projected/20986a23-c21b-4951-98be-b22b660f14b2-kube-api-access-4tq6w\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:00.826642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.826551 5633 generic.go:356] "Generic (PLEG): container finished" podID="20986a23-c21b-4951-98be-b22b660f14b2" containerID="21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022" exitCode=0 Nov 01 21:16:00.826642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.826599 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" Nov 01 21:16:00.826857 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.826638 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" event={"ID":"20986a23-c21b-4951-98be-b22b660f14b2","Type":"ContainerDied","Data":"21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022"} Nov 01 21:16:00.826857 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.826691 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph" event={"ID":"20986a23-c21b-4951-98be-b22b660f14b2","Type":"ContainerDied","Data":"95f4ef267954107da19d9c94b5d9043fe3516eb48a4cc08f6c1857c9411163f3"} Nov 01 21:16:00.826857 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.826713 5633 scope.go:122] "RemoveContainer" containerID="21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022" Nov 01 21:16:00.836444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.836424 5633 scope.go:122] "RemoveContainer" containerID="21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022" Nov 01 21:16:00.836769 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:16:00.836750 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022\": container with ID starting with 21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022 not found: ID does not exist" containerID="21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022" Nov 01 21:16:00.836818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.836782 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022"} err="failed to get container status \"21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022\": rpc error: code = NotFound desc = could not find container \"21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022\": container with ID starting with 21e7e2767c91f14ea598469bfabdd576261d8ac0518671e071ac0a5b3717e022 not found: ID does not exist" Nov 01 21:16:00.990116 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.990074 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph"] Nov 01 21:16:00.994311 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:00.994286 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6a675a-69649f8bd7-bscph"] Nov 01 21:16:01.771548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:01.771509 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:02.345664 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:02.345623 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:02.466216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:02.466181 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20986a23-c21b-4951-98be-b22b660f14b2" path="/var/lib/kubelet/pods/20986a23-c21b-4951-98be-b22b660f14b2/volumes" Nov 01 21:16:02.617950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:02.617863 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:05.529971 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:05.529936 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:05.668406 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:05.668355 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:05.789160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:05.789061 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:08.735538 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:08.735494 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:08.896135 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:08.896100 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:09.729375 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:09.729312 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:09.920977 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:09.920943 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:11.808324 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:11.808290 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:11.968298 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:11.968261 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:12.086285 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:12.086195 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:16.156802 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:16:16.156761 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:16:16.675154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:16.675120 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:19.683443 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:19.683409 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:16:21.377642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:21.377592 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:26.073800 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:26.073767 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:30.971685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:30.971645 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:36.269318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:36.269269 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:40.975529 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:40.975489 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:41.804633 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.804595 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl"] Nov 01 21:16:41.805034 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805021 5633 cpu_manager.go:414] "containerMap: removing container" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:16:41.805083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805037 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:16:41.805083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805054 5633 cpu_manager.go:414] "containerMap: removing container" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" Nov 01 21:16:41.805083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805059 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" Nov 01 21:16:41.805083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805066 5633 cpu_manager.go:414] "containerMap: removing container" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerName="payload-processing" Nov 01 21:16:41.805083 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805071 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerName="payload-processing" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805089 5633 cpu_manager.go:414] "containerMap: removing container" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805094 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805105 5633 cpu_manager.go:414] "containerMap: removing container" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805109 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805117 5633 cpu_manager.go:414] "containerMap: removing container" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805122 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805137 5633 cpu_manager.go:414] "containerMap: removing container" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805142 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805148 5633 cpu_manager.go:414] "containerMap: removing container" podUID="20986a23-c21b-4951-98be-b22b660f14b2" containerName="payload-pre-processing" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805153 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20986a23-c21b-4951-98be-b22b660f14b2" containerName="payload-pre-processing" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805218 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805226 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805232 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805238 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="20986a23-c21b-4951-98be-b22b660f14b2" containerName="payload-pre-processing" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805244 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:16:41.805246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805251 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" Nov 01 21:16:41.805785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805256 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="02897cac-e8f4-42b8-a51a-4e09aa1b54f0" containerName="payload-processing" Nov 01 21:16:41.805785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805263 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805268 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.805785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.805273 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="86414fcd-5b69-4886-9a0a-4fa0c733b9f3" containerName="cleanup" Nov 01 21:16:41.808634 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.808616 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.861114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.861076 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.861325 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.861128 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.861325 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.861263 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwxrc\" (UniqueName: \"kubernetes.io/projected/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-kube-api-access-lwxrc\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.908112 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.908078 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:16:41.910105 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.910074 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:16:41.910268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.910115 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:16:41.910268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.910185 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:16:41.911310 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.911290 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl"] Nov 01 21:16:41.962550 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.962513 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.962779 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.962567 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxrc\" (UniqueName: \"kubernetes.io/projected/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-kube-api-access-lwxrc\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.962779 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.962667 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.963233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.963207 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:41.965171 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:41.965148 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:42.019529 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:42.019495 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxrc\" (UniqueName: \"kubernetes.io/projected/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-kube-api-access-lwxrc\") pod \"maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:42.122477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:42.122387 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:42.498514 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:42.498483 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl"] Nov 01 21:16:42.499621 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:16:42.499592 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7340fb4_30cc_4fe2_9da3_fbb64cced0c4.slice/crio-a482ba0b89401194cca20bd7029f52265c74888dd1e46f65d8d73185bab7641c WatchSource:0}: Error finding container a482ba0b89401194cca20bd7029f52265c74888dd1e46f65d8d73185bab7641c: Status 404 returned error can't find the container with id a482ba0b89401194cca20bd7029f52265c74888dd1e46f65d8d73185bab7641c Nov 01 21:16:42.533438 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:42.533390 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:43.078702 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:43.078606 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" event={"ID":"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4","Type":"ContainerStarted","Data":"8f7777d2595c966829b779fddb1350a33759a532bd493c9b913e719e0f424ee7"} Nov 01 21:16:43.078702 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:43.078660 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" event={"ID":"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4","Type":"ContainerStarted","Data":"a482ba0b89401194cca20bd7029f52265c74888dd1e46f65d8d73185bab7641c"} Nov 01 21:16:43.192999 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:43.192944 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" podStartSLOduration=2.192930232 podStartE2EDuration="2.192930232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:43.190426488 +0000 UTC m=+4410.864685744" watchObservedRunningTime="2026-11-01 21:16:43.192930232 +0000 UTC m=+4410.867189494" Nov 01 21:16:44.085379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:44.085318 5633 generic.go:356] "Generic (PLEG): container finished" podID="a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" containerID="8f7777d2595c966829b779fddb1350a33759a532bd493c9b913e719e0f424ee7" exitCode=0 Nov 01 21:16:44.085379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:44.085367 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" event={"ID":"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4","Type":"ContainerDied","Data":"8f7777d2595c966829b779fddb1350a33759a532bd493c9b913e719e0f424ee7"} Nov 01 21:16:45.236096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.236069 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:45.292588 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.292553 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-xks-ca\") pod \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " Nov 01 21:16:45.292772 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.292647 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-kube-api-access-lwxrc\" (UniqueName: \"kubernetes.io/projected/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-kube-api-access-lwxrc\") pod \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " Nov 01 21:16:45.292772 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.292729 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-openshift-ca\") pod \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\" (UID: \"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4\") " Nov 01 21:16:45.293092 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.293067 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-openshift-ca" pod "a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" (UID: "a7340fb4-30cc-4fe2-9da3-fbb64cced0c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:16:45.295085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.295050 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-xks-ca" pod "a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" (UID: "a7340fb4-30cc-4fe2-9da3-fbb64cced0c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:16:45.295199 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.295065 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-kube-api-access-lwxrc" pod "a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" (UID: "a7340fb4-30cc-4fe2-9da3-fbb64cced0c4"). InnerVolumeSpecName "kube-api-access-lwxrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:16:45.393692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.393611 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwxrc\" (UniqueName: \"kubernetes.io/projected/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-kube-api-access-lwxrc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:45.393692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.393641 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:45.393692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.393652 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:16:45.673715 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:45.673607 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:46.098056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:46.098024 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" event={"ID":"a7340fb4-30cc-4fe2-9da3-fbb64cced0c4","Type":"ContainerDied","Data":"a482ba0b89401194cca20bd7029f52265c74888dd1e46f65d8d73185bab7641c"} Nov 01 21:16:46.098252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:46.098062 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl" Nov 01 21:16:46.098252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:46.098072 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a482ba0b89401194cca20bd7029f52265c74888dd1e46f65d8d73185bab7641c" Nov 01 21:16:46.155848 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:16:46.155814 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:16:50.376517 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:50.376474 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:53.614658 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:53.614566 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:53.744037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:53.743997 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:55.078689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:55.078652 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:16:59.179422 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:16:59.179348 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:00.650141 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:00.650104 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:01.035307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:01.035260 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:01.147568 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:01.147526 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl"] Nov 01 21:17:01.197021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:01.196973 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e843fb4-8nhdl"] Nov 01 21:17:02.466872 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:02.466837 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" path="/var/lib/kubelet/pods/a7340fb4-30cc-4fe2-9da3-fbb64cced0c4/volumes" Nov 01 21:17:02.788091 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:02.788004 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:05.294559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:05.294525 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:17:07.090556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.090519 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-mc6zb"] Nov 01 21:17:07.091051 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.090729 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerName="manager" containerID="cri-o://64128474a9e8753156c095bfcf333a6079670019c7a0aacb166de84fb457ea95" gracePeriod=30 Nov 01 21:17:07.219068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.219036 5633 generic.go:356] "Generic (PLEG): container finished" podID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerID="64128474a9e8753156c095bfcf333a6079670019c7a0aacb166de84fb457ea95" exitCode=0 Nov 01 21:17:07.219068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.219076 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" event={"ID":"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864","Type":"ContainerDied","Data":"64128474a9e8753156c095bfcf333a6079670019c7a0aacb166de84fb457ea95"} Nov 01 21:17:07.219321 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.219138 5633 scope.go:122] "RemoveContainer" containerID="4b045db74f95217d8eefbf1aa5ac43bfe18d842741a9146a7067267a30a7de9e" Nov 01 21:17:07.348489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.348424 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 21:17:07.511206 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.511158 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-webhook-cert\") pod \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " Nov 01 21:17:07.511404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.511313 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-kube-api-access-pwctv\" (UniqueName: \"kubernetes.io/projected/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-kube-api-access-pwctv\") pod \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " Nov 01 21:17:07.511464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.511426 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-metrics-certs\") pod \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\" (UID: \"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864\") " Nov 01 21:17:07.513709 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.513673 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-webhook-cert" pod "f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" (UID: "f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:17:07.513868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.513717 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-metrics-certs" pod "f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" (UID: "f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:17:07.513868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.513810 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-kube-api-access-pwctv" pod "f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" (UID: "f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864"). InnerVolumeSpecName "kube-api-access-pwctv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:17:07.612894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.612806 5633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-metrics-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:07.612894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.612836 5633 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-webhook-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:07.612894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:07.612850 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwctv\" (UniqueName: \"kubernetes.io/projected/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864-kube-api-access-pwctv\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:08.225827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:08.225797 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" Nov 01 21:17:08.226357 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:08.225823 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-mc6zb" event={"ID":"f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864","Type":"ContainerDied","Data":"ac957764179a61eaa3dcb395d0f241e822ab6e264acb799fcbdb70f0f181566f"} Nov 01 21:17:08.226357 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:08.225894 5633 scope.go:122] "RemoveContainer" containerID="64128474a9e8753156c095bfcf333a6079670019c7a0aacb166de84fb457ea95" Nov 01 21:17:08.388401 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:08.388355 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-mc6zb"] Nov 01 21:17:08.437273 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:08.437238 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-mc6zb"] Nov 01 21:17:08.466241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:08.466203 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" path="/var/lib/kubelet/pods/f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864/volumes" Nov 01 21:17:09.194646 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.194610 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645c58c6f7-dh87m"] Nov 01 21:17:09.195126 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195106 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerName="manager" Nov 01 21:17:09.195232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195130 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerName="manager" Nov 01 21:17:09.195232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195163 5633 cpu_manager.go:414] "containerMap: removing container" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:17:09.195232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195171 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35103c1f-a5da-47b0-a0fc-8e834459cd06" containerName="cleanup" Nov 01 21:17:09.195232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195193 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" containerName="revoke-keys" Nov 01 21:17:09.195232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195201 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" containerName="revoke-keys" Nov 01 21:17:09.195232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195218 5633 cpu_manager.go:414] "containerMap: removing container" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" Nov 01 21:17:09.195232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195225 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ede63f-d465-4d69-bda7-3800a6f7d614" containerName="istio-proxy" Nov 01 21:17:09.195659 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195355 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7340fb4-30cc-4fe2-9da3-fbb64cced0c4" containerName="revoke-keys" Nov 01 21:17:09.195659 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.195369 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerName="manager" Nov 01 21:17:09.200145 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.200123 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.254047 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.254014 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-dh87m"] Nov 01 21:17:09.299820 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.299781 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 21:17:09.302045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.302018 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 21:17:09.302045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.302021 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 21:17:09.302250 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.302025 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5ch2z\"" Nov 01 21:17:09.332159 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.332127 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4f4p\" (UniqueName: \"kubernetes.io/projected/9b0f464b-b7bf-4246-b0f7-0652743c21cc-kube-api-access-b4f4p\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.332317 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.332292 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-webhook-cert\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.332433 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.332364 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-metrics-certs\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.432991 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.432945 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-webhook-cert\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.432991 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.432992 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-metrics-certs\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.433230 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.433024 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4f4p\" (UniqueName: \"kubernetes.io/projected/9b0f464b-b7bf-4246-b0f7-0652743c21cc-kube-api-access-b4f4p\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.435742 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.435711 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-metrics-certs\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.435742 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.435741 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-webhook-cert\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.489189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.489151 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4f4p\" (UniqueName: \"kubernetes.io/projected/9b0f464b-b7bf-4246-b0f7-0652743c21cc-kube-api-access-b4f4p\") pod \"maas-controller-645c58c6f7-dh87m\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.513062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.513025 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:09.887982 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.887950 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-dh87m"] Nov 01 21:17:09.889175 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:17:09.889149 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0f464b_b7bf_4246_b0f7_0652743c21cc.slice/crio-b05a4ee215d80d1776c80d29ceb624aeca3cec9d6bc5624724b7450117afaf54 WatchSource:0}: Error finding container b05a4ee215d80d1776c80d29ceb624aeca3cec9d6bc5624724b7450117afaf54: Status 404 returned error can't find the container with id b05a4ee215d80d1776c80d29ceb624aeca3cec9d6bc5624724b7450117afaf54 Nov 01 21:17:09.890487 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:09.890465 5633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:17:10.240523 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:10.240480 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-dh87m" event={"ID":"9b0f464b-b7bf-4246-b0f7-0652743c21cc","Type":"ContainerStarted","Data":"b05a4ee215d80d1776c80d29ceb624aeca3cec9d6bc5624724b7450117afaf54"} Nov 01 21:17:11.247312 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:11.247279 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-dh87m" event={"ID":"9b0f464b-b7bf-4246-b0f7-0652743c21cc","Type":"ContainerStarted","Data":"81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4"} Nov 01 21:17:11.247744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:11.247408 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:11.361341 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:11.361277 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-645c58c6f7-dh87m" podStartSLOduration=1.422427735 podStartE2EDuration="2.361259878s" totalImagesPullingTime="938.832143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:11.360811227 +0000 UTC m=+4439.035070497" watchObservedRunningTime="2026-11-01 21:17:11.361259878 +0000 UTC m=+4439.035519134" Nov 01 21:17:16.156714 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:17:16.156666 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:17:22.255064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:22.255024 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:30.916389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:30.916324 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf"] Nov 01 21:17:30.916815 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:30.916801 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerName="manager" Nov 01 21:17:30.916861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:30.916816 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerName="manager" Nov 01 21:17:30.916914 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:30.916905 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ae1fa9-afa7-4162-b9e7-28bf4d6c2864" containerName="manager" Nov 01 21:17:30.920120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:30.920103 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:30.976089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:30.976057 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf"] Nov 01 21:17:31.021648 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.021616 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:17:31.021858 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.021652 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:17:31.021858 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.021650 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:17:31.021858 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.021739 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:17:31.040488 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.040445 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vbl\" (UniqueName: \"kubernetes.io/projected/e2c23098-bf82-4db0-8156-2cb96d4ffeba-kube-api-access-c7vbl\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.040684 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.040528 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.040684 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.040635 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.141345 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.141282 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vbl\" (UniqueName: \"kubernetes.io/projected/e2c23098-bf82-4db0-8156-2cb96d4ffeba-kube-api-access-c7vbl\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.141572 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.141389 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.141572 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.141429 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.142211 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.142183 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.144050 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.144021 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.198123 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.198037 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vbl\" (UniqueName: \"kubernetes.io/projected/e2c23098-bf82-4db0-8156-2cb96d4ffeba-kube-api-access-c7vbl\") pod \"maas-api-revoke-sub-85237e45-pfdxf\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.234036 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.233989 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:31.613387 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:31.613350 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf"] Nov 01 21:17:31.614462 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:17:31.614429 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c23098_bf82_4db0_8156_2cb96d4ffeba.slice/crio-e1a8617b879b376511477b59aafe08498141fc47e9d57404420a3530658d654e WatchSource:0}: Error finding container e1a8617b879b376511477b59aafe08498141fc47e9d57404420a3530658d654e: Status 404 returned error can't find the container with id e1a8617b879b376511477b59aafe08498141fc47e9d57404420a3530658d654e Nov 01 21:17:32.370027 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:32.369992 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" event={"ID":"e2c23098-bf82-4db0-8156-2cb96d4ffeba","Type":"ContainerStarted","Data":"4b12a847ede21fd1ff0ae132bc0d257439ab5a664c166e1b89e05f1635b7b77b"} Nov 01 21:17:32.370442 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:32.370041 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" event={"ID":"e2c23098-bf82-4db0-8156-2cb96d4ffeba","Type":"ContainerStarted","Data":"e1a8617b879b376511477b59aafe08498141fc47e9d57404420a3530658d654e"} Nov 01 21:17:32.483906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:32.483854 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" podStartSLOduration=2.483841386 podStartE2EDuration="2.483841386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:32.480687574 +0000 UTC m=+4460.154946828" watchObservedRunningTime="2026-11-01 21:17:32.483841386 +0000 UTC m=+4460.158100683" Nov 01 21:17:33.376820 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:33.376783 5633 generic.go:356] "Generic (PLEG): container finished" podID="e2c23098-bf82-4db0-8156-2cb96d4ffeba" containerID="4b12a847ede21fd1ff0ae132bc0d257439ab5a664c166e1b89e05f1635b7b77b" exitCode=0 Nov 01 21:17:33.377309 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:33.376869 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" event={"ID":"e2c23098-bf82-4db0-8156-2cb96d4ffeba","Type":"ContainerDied","Data":"4b12a847ede21fd1ff0ae132bc0d257439ab5a664c166e1b89e05f1635b7b77b"} Nov 01 21:17:34.527310 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.527282 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:34.575063 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.575022 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-xks-ca\") pod \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " Nov 01 21:17:34.575269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.575084 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c23098-bf82-4db0-8156-2cb96d4ffeba-kube-api-access-c7vbl\" (UniqueName: \"kubernetes.io/projected/e2c23098-bf82-4db0-8156-2cb96d4ffeba-kube-api-access-c7vbl\") pod \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " Nov 01 21:17:34.575269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.575229 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-openshift-ca\") pod \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\" (UID: \"e2c23098-bf82-4db0-8156-2cb96d4ffeba\") " Nov 01 21:17:34.575585 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.575561 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-openshift-ca" pod "e2c23098-bf82-4db0-8156-2cb96d4ffeba" (UID: "e2c23098-bf82-4db0-8156-2cb96d4ffeba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:17:34.577247 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.577226 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-xks-ca" pod "e2c23098-bf82-4db0-8156-2cb96d4ffeba" (UID: "e2c23098-bf82-4db0-8156-2cb96d4ffeba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:17:34.577417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.577398 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c23098-bf82-4db0-8156-2cb96d4ffeba-kube-api-access-c7vbl" pod "e2c23098-bf82-4db0-8156-2cb96d4ffeba" (UID: "e2c23098-bf82-4db0-8156-2cb96d4ffeba"). InnerVolumeSpecName "kube-api-access-c7vbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:17:34.676819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.676731 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vbl\" (UniqueName: \"kubernetes.io/projected/e2c23098-bf82-4db0-8156-2cb96d4ffeba-kube-api-access-c7vbl\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:34.676819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.676766 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:34.676819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:34.676781 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c23098-bf82-4db0-8156-2cb96d4ffeba-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:35.390987 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:35.390954 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" Nov 01 21:17:35.391213 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:35.390958 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf" event={"ID":"e2c23098-bf82-4db0-8156-2cb96d4ffeba","Type":"ContainerDied","Data":"e1a8617b879b376511477b59aafe08498141fc47e9d57404420a3530658d654e"} Nov 01 21:17:35.391213 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:35.391102 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a8617b879b376511477b59aafe08498141fc47e9d57404420a3530658d654e" Nov 01 21:17:36.211287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.211244 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc"] Nov 01 21:17:36.211937 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.211914 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e2c23098-bf82-4db0-8156-2cb96d4ffeba" containerName="revoke-keys" Nov 01 21:17:36.212036 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.211941 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c23098-bf82-4db0-8156-2cb96d4ffeba" containerName="revoke-keys" Nov 01 21:17:36.212095 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.212084 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c23098-bf82-4db0-8156-2cb96d4ffeba" containerName="revoke-keys" Nov 01 21:17:36.215975 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.215956 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.274628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.274591 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 21:17:36.290234 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290203 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290405 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290264 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290405 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290365 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rkh\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-kube-api-access-s6rkh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290488 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290460 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290512 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290564 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290542 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290655 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290640 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290668 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.290732 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.290698 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.322213 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.322175 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\"" Nov 01 21:17:36.325580 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.325556 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc"] Nov 01 21:17:36.391737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391698 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.391737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391735 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391762 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391807 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391827 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391851 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rkh\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-kube-api-access-s6rkh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391897 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391940 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.391981 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.392168 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.392175 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.392232 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.392739 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.392705 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.393014 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.392884 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.394666 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.394641 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.394757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.394702 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.448114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.448077 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.448295 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.448218 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rkh\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-kube-api-access-s6rkh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fjrwc\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.529598 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.529512 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:36.957410 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.957382 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc"] Nov 01 21:17:36.962317 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:17:36.961924 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ce3ee8_0815_4096_ad3b_f886a79b7032.slice/crio-0afc661e503909678aafe3bebc0dc9448e68a55293b3c5da428f800c0135f09d WatchSource:0}: Error finding container 0afc661e503909678aafe3bebc0dc9448e68a55293b3c5da428f800c0135f09d: Status 404 returned error can't find the container with id 0afc661e503909678aafe3bebc0dc9448e68a55293b3c5da428f800c0135f09d Nov 01 21:17:36.963825 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.963793 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:17:36.963920 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.963878 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:17:36.963920 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:36.963909 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:17:37.404408 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:37.404376 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" event={"ID":"f7ce3ee8-0815-4096-ad3b-f886a79b7032","Type":"ContainerStarted","Data":"db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e"} Nov 01 21:17:37.404838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:37.404441 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" event={"ID":"f7ce3ee8-0815-4096-ad3b-f886a79b7032","Type":"ContainerStarted","Data":"0afc661e503909678aafe3bebc0dc9448e68a55293b3c5da428f800c0135f09d"} Nov 01 21:17:37.530604 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:37.530567 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:38.531434 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:38.531403 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 01 21:17:38.531944 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:38.531472 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 01 21:17:39.530513 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:39.530478 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 01 21:17:39.530710 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:39.530542 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 01 21:17:40.530772 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:40.530733 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 01 21:17:40.531268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:40.530796 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 01 21:17:41.530618 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:41.530580 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 01 21:17:41.530847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:41.530643 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 01 21:17:42.530038 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:42.529994 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 01 21:17:42.530280 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:42.530049 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 01 21:17:42.970891 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:42.970860 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:43.407166 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.407057 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podStartSLOduration=7.407040654 podStartE2EDuration="7.407040654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:37.617252222 +0000 UTC m=+4465.291511477" watchObservedRunningTime="2026-11-01 21:17:43.407040654 +0000 UTC m=+4471.081299908" Nov 01 21:17:43.408748 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.408723 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-dh87m"] Nov 01 21:17:43.408949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.408928 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-645c58c6f7-dh87m" podUID="9b0f464b-b7bf-4246-b0f7-0652743c21cc" containerName="manager" containerID="cri-o://81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4" gracePeriod=30 Nov 01 21:17:43.530863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.530831 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 01 21:17:43.531001 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.530881 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 01 21:17:43.666713 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.666647 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:43.784410 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.784379 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-metrics-certs\") pod \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " Nov 01 21:17:43.784606 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.784436 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-webhook-cert\") pod \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " Nov 01 21:17:43.784606 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.784471 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0f464b-b7bf-4246-b0f7-0652743c21cc-kube-api-access-b4f4p\" (UniqueName: \"kubernetes.io/projected/9b0f464b-b7bf-4246-b0f7-0652743c21cc-kube-api-access-b4f4p\") pod \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\" (UID: \"9b0f464b-b7bf-4246-b0f7-0652743c21cc\") " Nov 01 21:17:43.787807 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.787509 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-webhook-cert" pod "9b0f464b-b7bf-4246-b0f7-0652743c21cc" (UID: "9b0f464b-b7bf-4246-b0f7-0652743c21cc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:17:43.788477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.788435 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0f464b-b7bf-4246-b0f7-0652743c21cc-kube-api-access-b4f4p" pod "9b0f464b-b7bf-4246-b0f7-0652743c21cc" (UID: "9b0f464b-b7bf-4246-b0f7-0652743c21cc"). InnerVolumeSpecName "kube-api-access-b4f4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:17:43.789357 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.789295 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-metrics-certs" pod "9b0f464b-b7bf-4246-b0f7-0652743c21cc" (UID: "9b0f464b-b7bf-4246-b0f7-0652743c21cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:17:43.885706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.885671 5633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-metrics-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:43.885706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.885703 5633 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b0f464b-b7bf-4246-b0f7-0652743c21cc-webhook-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:43.885970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:43.885716 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4f4p\" (UniqueName: \"kubernetes.io/projected/9b0f464b-b7bf-4246-b0f7-0652743c21cc-kube-api-access-b4f4p\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:17:44.361761 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.361718 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:44.454044 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.454015 5633 generic.go:356] "Generic (PLEG): container finished" podID="9b0f464b-b7bf-4246-b0f7-0652743c21cc" containerID="81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4" exitCode=0 Nov 01 21:17:44.454257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.454079 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-dh87m" Nov 01 21:17:44.454257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.454092 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-dh87m" event={"ID":"9b0f464b-b7bf-4246-b0f7-0652743c21cc","Type":"ContainerDied","Data":"81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4"} Nov 01 21:17:44.454257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.454146 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-dh87m" event={"ID":"9b0f464b-b7bf-4246-b0f7-0652743c21cc","Type":"ContainerDied","Data":"b05a4ee215d80d1776c80d29ceb624aeca3cec9d6bc5624724b7450117afaf54"} Nov 01 21:17:44.454257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.454169 5633 scope.go:122] "RemoveContainer" containerID="81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4" Nov 01 21:17:44.464966 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.464921 5633 scope.go:122] "RemoveContainer" containerID="81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4" Nov 01 21:17:44.465314 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:17:44.465284 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4\": container with ID starting with 81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4 not found: ID does not exist" containerID="81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4" Nov 01 21:17:44.465461 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.465325 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4"} err="failed to get container status \"81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4\": rpc error: code = NotFound desc = could not find container \"81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4\": container with ID starting with 81665249d2ec3590da8b84b51afb877d0fbb5488162e1368b2276cbaef1ac4e4 not found: ID does not exist" Nov 01 21:17:44.530706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.530667 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 01 21:17:44.530896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.530737 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 01 21:17:44.619113 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.619029 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-dh87m"] Nov 01 21:17:44.623794 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:44.623768 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-dh87m"] Nov 01 21:17:45.867855 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:45.867820 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:46.011611 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.011576 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645c58c6f7-fvkrq"] Nov 01 21:17:46.012117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.012098 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9b0f464b-b7bf-4246-b0f7-0652743c21cc" containerName="manager" Nov 01 21:17:46.012232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.012119 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0f464b-b7bf-4246-b0f7-0652743c21cc" containerName="manager" Nov 01 21:17:46.012297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.012247 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0f464b-b7bf-4246-b0f7-0652743c21cc" containerName="manager" Nov 01 21:17:46.025860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.025831 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.113204 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.113166 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzxm\" (UniqueName: \"kubernetes.io/projected/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-kube-api-access-jnzxm\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.113436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.113224 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-webhook-cert\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.113436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.113290 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-metrics-certs\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.127244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.127106 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 21:17:46.127244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.127197 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 21:17:46.127244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.127164 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 21:17:46.127244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.127171 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5ch2z\"" Nov 01 21:17:46.132622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.132589 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-fvkrq"] Nov 01 21:17:46.157312 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:17:46.157272 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:17:46.214163 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.214127 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-metrics-certs\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.214418 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.214254 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzxm\" (UniqueName: \"kubernetes.io/projected/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-kube-api-access-jnzxm\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.214418 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.214292 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-webhook-cert\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.217072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.217050 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-webhook-cert\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.217196 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.217130 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-metrics-certs\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.271089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.271052 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzxm\" (UniqueName: \"kubernetes.io/projected/3249de5b-b781-4eeb-8fe0-9bf4455e9af1-kube-api-access-jnzxm\") pod \"maas-controller-645c58c6f7-fvkrq\" (UID: \"3249de5b-b781-4eeb-8fe0-9bf4455e9af1\") " pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.343059 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.343019 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:46.399715 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.399639 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:46.400648 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.400617 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:46.401722 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.401644 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:17:46.466159 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.466124 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0f464b-b7bf-4246-b0f7-0652743c21cc" path="/var/lib/kubelet/pods/9b0f464b-b7bf-4246-b0f7-0652743c21cc/volumes" Nov 01 21:17:46.742312 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:46.742282 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645c58c6f7-fvkrq"] Nov 01 21:17:46.743235 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:17:46.743211 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3249de5b_b781_4eeb_8fe0_9bf4455e9af1.slice/crio-aa518e0ae78125bc85b83cffba04ac9cc14c81854e92d4bb1b33e6f60859c9ed WatchSource:0}: Error finding container aa518e0ae78125bc85b83cffba04ac9cc14c81854e92d4bb1b33e6f60859c9ed: Status 404 returned error can't find the container with id aa518e0ae78125bc85b83cffba04ac9cc14c81854e92d4bb1b33e6f60859c9ed Nov 01 21:17:47.477281 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:47.477243 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-fvkrq" event={"ID":"3249de5b-b781-4eeb-8fe0-9bf4455e9af1","Type":"ContainerStarted","Data":"aa518e0ae78125bc85b83cffba04ac9cc14c81854e92d4bb1b33e6f60859c9ed"} Nov 01 21:17:48.484074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:48.484038 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645c58c6f7-fvkrq" event={"ID":"3249de5b-b781-4eeb-8fe0-9bf4455e9af1","Type":"ContainerStarted","Data":"72f6d07176bed85e85fe1ed5c2866a21277f9277cc02187abb5ea3be796871ed"} Nov 01 21:17:48.484590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:48.484100 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:48.597561 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:48.597511 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-645c58c6f7-fvkrq" podStartSLOduration=2.663964739 podStartE2EDuration="3.597490316s" totalImagesPullingTime="933.525577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:48.596470355 +0000 UTC m=+4476.270729610" watchObservedRunningTime="2026-11-01 21:17:48.597490316 +0000 UTC m=+4476.271749571" Nov 01 21:17:51.217262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.217221 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c"] Nov 01 21:17:51.221028 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.221009 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.321728 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.321697 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c"] Nov 01 21:17:51.364811 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.364767 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78pgb\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-kube-api-access-78pgb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.364985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.364817 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.364985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.364930 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.365110 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.365023 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.365110 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.365078 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.365214 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.365123 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.365214 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.365144 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.365375 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.365210 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.365375 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.365284 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466770 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466729 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466792 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466818 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466836 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466860 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466888 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466911 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78pgb\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-kube-api-access-78pgb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.466943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466936 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.467489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.466976 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.467489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.467288 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.467489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.467310 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.467489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.467405 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.467489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.467460 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.467965 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.467831 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.469385 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.469361 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.469587 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.469568 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.523208 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.523170 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.523434 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.523309 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78pgb\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-kube-api-access-78pgb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9rk4c\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.534254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.534226 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:51.963609 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.963576 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c"] Nov 01 21:17:51.964532 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:17:51.964499 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb661e9d8_10bf_490d_9ca9_1093dca1ddc1.slice/crio-465aef1ed890de9ca17eedfbdc23da4a34d3b36d037711ad98ac40e4932316fe WatchSource:0}: Error finding container 465aef1ed890de9ca17eedfbdc23da4a34d3b36d037711ad98ac40e4932316fe: Status 404 returned error can't find the container with id 465aef1ed890de9ca17eedfbdc23da4a34d3b36d037711ad98ac40e4932316fe Nov 01 21:17:51.966828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.966791 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:17:51.966922 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.966862 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:17:51.966922 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:51.966896 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:17:52.079166 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:52.079120 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:17:52.516194 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:52.516153 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" event={"ID":"b661e9d8-10bf-490d-9ca9-1093dca1ddc1","Type":"ContainerStarted","Data":"c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd"} Nov 01 21:17:52.516646 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:52.516204 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" event={"ID":"b661e9d8-10bf-490d-9ca9-1093dca1ddc1","Type":"ContainerStarted","Data":"465aef1ed890de9ca17eedfbdc23da4a34d3b36d037711ad98ac40e4932316fe"} Nov 01 21:17:52.535081 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:52.535052 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:17:52.630012 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:52.629942 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podStartSLOduration=1.629923051 podStartE2EDuration="1.629923051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:52.62983893 +0000 UTC m=+4480.304098198" watchObservedRunningTime="2026-11-01 21:17:52.629923051 +0000 UTC m=+4480.304182301" Nov 01 21:17:53.536906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:53.536863 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 21:17:53.537369 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:53.536923 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 21:17:54.534644 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:54.534605 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 21:17:54.534848 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:54.534665 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 21:17:55.534685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:55.534644 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 21:17:55.535178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:55.534698 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 21:17:56.534782 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:56.534744 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 21:17:56.535201 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:56.534806 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 21:17:57.535254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:57.535210 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 21:17:57.535788 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:57.535273 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 21:17:58.535323 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:58.535283 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 21:17:58.535769 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:58.535377 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 21:17:59.492180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:59.492149 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645c58c6f7-fvkrq" Nov 01 21:17:59.535285 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:59.535201 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 01 21:17:59.535285 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:17:59.535273 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 01 21:18:01.300608 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:01.300558 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:18:01.301091 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:01.300879 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:18:01.301938 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:01.301918 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:18:12.900997 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:12.900947 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:18:16.164814 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:18:16.164779 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:18:20.425777 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:20.425745 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:18:31.964533 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:31.964450 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:18:36.580817 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.580780 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t"] Nov 01 21:18:36.584932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.584910 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.677838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.677797 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbx8\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-kube-api-access-bmbx8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678051 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.677859 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678051 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.677964 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678051 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.678033 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678239 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.678134 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678239 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.678180 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8eae30-2493-4455-8868-4230de1d9526-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678366 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.678247 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8eae30-2493-4455-8868-4230de1d9526-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678366 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.678267 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.678461 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.678371 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779157 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbx8\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-kube-api-access-bmbx8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779215 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779245 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779299 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779386 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779422 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8eae30-2493-4455-8868-4230de1d9526-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779452 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8eae30-2493-4455-8868-4230de1d9526-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779477 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779570 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779536 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779883 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779855 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.779943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779919 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.780002 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.779977 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.780312 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.780266 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8eae30-2493-4455-8868-4230de1d9526-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.781489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.781462 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.782816 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.782793 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.783790 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.783687 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8eae30-2493-4455-8868-4230de1d9526-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.787465 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.787435 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t"] Nov 01 21:18:36.839056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.838961 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.839244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.839134 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbx8\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-kube-api-access-bmbx8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fvd8t\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:36.898177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:36.898132 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:37.325786 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.325658 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t"] Nov 01 21:18:37.326462 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:18:37.326419 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8eae30_2493_4455_8868_4230de1d9526.slice/crio-f9c2d7dd91e7cfe0df5134bab3ae312812e84a51fa0adb8eecf13790c7d1eebb WatchSource:0}: Error finding container f9c2d7dd91e7cfe0df5134bab3ae312812e84a51fa0adb8eecf13790c7d1eebb: Status 404 returned error can't find the container with id f9c2d7dd91e7cfe0df5134bab3ae312812e84a51fa0adb8eecf13790c7d1eebb Nov 01 21:18:37.328934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.328892 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:18:37.329056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.328985 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:18:37.329056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.329029 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:18:37.805361 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.802215 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" event={"ID":"3b8eae30-2493-4455-8868-4230de1d9526","Type":"ContainerStarted","Data":"48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d"} Nov 01 21:18:37.805361 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.802286 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" event={"ID":"3b8eae30-2493-4455-8868-4230de1d9526","Type":"ContainerStarted","Data":"f9c2d7dd91e7cfe0df5134bab3ae312812e84a51fa0adb8eecf13790c7d1eebb"} Nov 01 21:18:37.903357 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.901433 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:37.922917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:37.922836 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podStartSLOduration=1.922813825 podStartE2EDuration="1.922813825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:37.919818962 +0000 UTC m=+4525.594078216" watchObservedRunningTime="2026-11-01 21:18:37.922813825 +0000 UTC m=+4525.597073081" Nov 01 21:18:38.917486 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:38.917450 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 21:18:38.928512 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:38.928466 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 21:18:39.906358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:39.901421 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 21:18:39.906358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:39.901488 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 21:18:40.899818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:40.899775 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 21:18:40.900484 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:40.900449 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 21:18:41.910524 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:41.910482 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 21:18:41.910976 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:41.910556 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 21:18:42.370932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:42.370894 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:18:42.900823 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:42.900779 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 21:18:42.901170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:42.901129 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 21:18:43.044319 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:43.044263 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9"] Nov 01 21:18:43.063231 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:43.063191 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63bb825d-tg8d9"] Nov 01 21:18:43.397244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:43.397204 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:18:43.899523 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:43.899493 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 21:18:43.899784 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:43.899547 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 21:18:44.467843 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:44.467812 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fade02-8ea2-4e45-b384-58b26246797f" path="/var/lib/kubelet/pods/76fade02-8ea2-4e45-b384-58b26246797f/volumes" Nov 01 21:18:44.898926 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:44.898846 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Nov 01 21:18:44.898926 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:44.898897 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Nov 01 21:18:46.158158 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:18:46.158125 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:18:46.700561 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:46.700530 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:46.701103 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:46.701078 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:46.701872 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:46.701848 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:18:52.960757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:52.960724 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:18:54.497096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.496988 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg"] Nov 01 21:18:54.501015 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.500938 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.557034 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.556998 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg"] Nov 01 21:18:54.597395 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.597363 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.597580 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.597412 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.597580 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.597518 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhz7\" (UniqueName: \"kubernetes.io/projected/5db1c6d9-1e9c-4d45-855f-61de797cffa1-kube-api-access-glhz7\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.602199 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.602168 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:18:54.602388 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.602253 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:18:54.602388 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.602267 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:18:54.602388 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.602283 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:18:54.698933 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.698884 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.699125 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.698946 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.699125 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.698993 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhz7\" (UniqueName: \"kubernetes.io/projected/5db1c6d9-1e9c-4d45-855f-61de797cffa1-kube-api-access-glhz7\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.699751 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.699723 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.701454 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.701431 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.755192 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.755113 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhz7\" (UniqueName: \"kubernetes.io/projected/5db1c6d9-1e9c-4d45-855f-61de797cffa1-kube-api-access-glhz7\") pod \"maas-api-revoke-sub-f03629db-mn9dg\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:54.818245 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:54.818213 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:55.051630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:55.051529 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv"] Nov 01 21:18:55.054002 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:55.053975 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615eb5af-gtlrv"] Nov 01 21:18:55.191989 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:55.191953 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg"] Nov 01 21:18:55.195741 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:18:55.195712 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db1c6d9_1e9c_4d45_855f_61de797cffa1.slice/crio-c69bb56e4de927c293641d689cf9435097639655024d16b993f6fcd92fcbc199 WatchSource:0}: Error finding container c69bb56e4de927c293641d689cf9435097639655024d16b993f6fcd92fcbc199: Status 404 returned error can't find the container with id c69bb56e4de927c293641d689cf9435097639655024d16b993f6fcd92fcbc199 Nov 01 21:18:55.998658 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:55.998621 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" event={"ID":"5db1c6d9-1e9c-4d45-855f-61de797cffa1","Type":"ContainerStarted","Data":"3268ff3a970022320d0ad3e98caddd757ee5bb442726c67d8edabcb6a4fd1753"} Nov 01 21:18:55.999178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:55.998689 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" event={"ID":"5db1c6d9-1e9c-4d45-855f-61de797cffa1","Type":"ContainerStarted","Data":"c69bb56e4de927c293641d689cf9435097639655024d16b993f6fcd92fcbc199"} Nov 01 21:18:56.112286 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:56.112222 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" podStartSLOduration=2.112206467 podStartE2EDuration="2.112206467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:56.110597201 +0000 UTC m=+4543.784856459" watchObservedRunningTime="2026-11-01 21:18:56.112206467 +0000 UTC m=+4543.786465722" Nov 01 21:18:56.466071 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:56.466045 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1722d9d1-e4ae-41a4-b9eb-afa33dab4de0" path="/var/lib/kubelet/pods/1722d9d1-e4ae-41a4-b9eb-afa33dab4de0/volumes" Nov 01 21:18:57.007955 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:57.007914 5633 generic.go:356] "Generic (PLEG): container finished" podID="5db1c6d9-1e9c-4d45-855f-61de797cffa1" containerID="3268ff3a970022320d0ad3e98caddd757ee5bb442726c67d8edabcb6a4fd1753" exitCode=0 Nov 01 21:18:57.008439 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:57.007998 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" event={"ID":"5db1c6d9-1e9c-4d45-855f-61de797cffa1","Type":"ContainerDied","Data":"3268ff3a970022320d0ad3e98caddd757ee5bb442726c67d8edabcb6a4fd1753"} Nov 01 21:18:58.160459 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.160429 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:58.237051 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.237015 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-openshift-ca\") pod \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " Nov 01 21:18:58.237260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.237084 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db1c6d9-1e9c-4d45-855f-61de797cffa1-kube-api-access-glhz7\" (UniqueName: \"kubernetes.io/projected/5db1c6d9-1e9c-4d45-855f-61de797cffa1-kube-api-access-glhz7\") pod \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " Nov 01 21:18:58.237260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.237222 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-xks-ca\") pod \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\" (UID: \"5db1c6d9-1e9c-4d45-855f-61de797cffa1\") " Nov 01 21:18:58.237440 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.237407 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-openshift-ca" pod "5db1c6d9-1e9c-4d45-855f-61de797cffa1" (UID: "5db1c6d9-1e9c-4d45-855f-61de797cffa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:18:58.237629 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.237614 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:18:58.239393 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.239361 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-xks-ca" pod "5db1c6d9-1e9c-4d45-855f-61de797cffa1" (UID: "5db1c6d9-1e9c-4d45-855f-61de797cffa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:18:58.239521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.239443 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db1c6d9-1e9c-4d45-855f-61de797cffa1-kube-api-access-glhz7" pod "5db1c6d9-1e9c-4d45-855f-61de797cffa1" (UID: "5db1c6d9-1e9c-4d45-855f-61de797cffa1"). InnerVolumeSpecName "kube-api-access-glhz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:18:58.338316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.338240 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5db1c6d9-1e9c-4d45-855f-61de797cffa1-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:18:58.338316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:58.338277 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhz7\" (UniqueName: \"kubernetes.io/projected/5db1c6d9-1e9c-4d45-855f-61de797cffa1-kube-api-access-glhz7\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:18:59.020930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:59.020899 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" Nov 01 21:18:59.021142 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:59.020937 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg" event={"ID":"5db1c6d9-1e9c-4d45-855f-61de797cffa1","Type":"ContainerDied","Data":"c69bb56e4de927c293641d689cf9435097639655024d16b993f6fcd92fcbc199"} Nov 01 21:18:59.021142 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:18:59.020982 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c69bb56e4de927c293641d689cf9435097639655024d16b993f6fcd92fcbc199" Nov 01 21:19:01.502441 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:01.502401 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:19:14.067363 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:14.067102 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7"] Nov 01 21:19:14.158984 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:14.158943 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f82128b9-fj4m7"] Nov 01 21:19:14.469035 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:14.468996 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5" path="/var/lib/kubelet/pods/8f1d20fa-dbc8-4d55-9e86-1cdc23d12ff5/volumes" Nov 01 21:19:15.521535 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.521489 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67"] Nov 01 21:19:15.522040 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.522007 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5db1c6d9-1e9c-4d45-855f-61de797cffa1" containerName="revoke-keys" Nov 01 21:19:15.522040 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.522021 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db1c6d9-1e9c-4d45-855f-61de797cffa1" containerName="revoke-keys" Nov 01 21:19:15.522120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.522101 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db1c6d9-1e9c-4d45-855f-61de797cffa1" containerName="revoke-keys" Nov 01 21:19:15.525605 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.525582 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.581362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.581309 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67"] Nov 01 21:19:15.584446 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.584412 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.584624 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.584468 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.584688 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.584644 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cggn6\" (UniqueName: \"kubernetes.io/projected/fd24214a-dc9a-4a7b-804b-1e25408d8844-kube-api-access-cggn6\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.626784 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.626746 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:19:15.626963 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.626850 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:19:15.626963 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.626893 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:19:15.627164 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.627149 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:19:15.685818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.685779 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.686011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.685849 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.686011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.685906 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cggn6\" (UniqueName: \"kubernetes.io/projected/fd24214a-dc9a-4a7b-804b-1e25408d8844-kube-api-access-cggn6\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.686543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.686519 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.688410 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.688392 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.743133 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.743101 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cggn6\" (UniqueName: \"kubernetes.io/projected/fd24214a-dc9a-4a7b-804b-1e25408d8844-kube-api-access-cggn6\") pod \"maas-api-revoke-sub-0165d7d2-7wm67\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:15.839349 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:15.839243 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:16.156027 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:19:16.155940 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:19:16.218735 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:16.218705 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67"] Nov 01 21:19:16.220144 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:19:16.220112 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd24214a_dc9a_4a7b_804b_1e25408d8844.slice/crio-0e60239a1cedb51105858026ac2defa8b31f87edb1ff44a5a5980620eea4b46d WatchSource:0}: Error finding container 0e60239a1cedb51105858026ac2defa8b31f87edb1ff44a5a5980620eea4b46d: Status 404 returned error can't find the container with id 0e60239a1cedb51105858026ac2defa8b31f87edb1ff44a5a5980620eea4b46d Nov 01 21:19:17.163275 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:17.163238 5633 generic.go:356] "Generic (PLEG): container finished" podID="fd24214a-dc9a-4a7b-804b-1e25408d8844" containerID="bf0a2f3a002e510b2305fd06341e1d565ca434799ecb8bd03892a2e694ca38a7" exitCode=0 Nov 01 21:19:17.163725 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:17.163285 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" event={"ID":"fd24214a-dc9a-4a7b-804b-1e25408d8844","Type":"ContainerDied","Data":"bf0a2f3a002e510b2305fd06341e1d565ca434799ecb8bd03892a2e694ca38a7"} Nov 01 21:19:17.163725 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:17.163322 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" event={"ID":"fd24214a-dc9a-4a7b-804b-1e25408d8844","Type":"ContainerStarted","Data":"0e60239a1cedb51105858026ac2defa8b31f87edb1ff44a5a5980620eea4b46d"} Nov 01 21:19:18.321084 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.321059 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:18.416822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.416777 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd24214a-dc9a-4a7b-804b-1e25408d8844-kube-api-access-cggn6\" (UniqueName: \"kubernetes.io/projected/fd24214a-dc9a-4a7b-804b-1e25408d8844-kube-api-access-cggn6\") pod \"fd24214a-dc9a-4a7b-804b-1e25408d8844\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " Nov 01 21:19:18.417055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.416882 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-xks-ca\") pod \"fd24214a-dc9a-4a7b-804b-1e25408d8844\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " Nov 01 21:19:18.417055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.416992 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-openshift-ca\") pod \"fd24214a-dc9a-4a7b-804b-1e25408d8844\" (UID: \"fd24214a-dc9a-4a7b-804b-1e25408d8844\") " Nov 01 21:19:18.417394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.417366 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-openshift-ca" pod "fd24214a-dc9a-4a7b-804b-1e25408d8844" (UID: "fd24214a-dc9a-4a7b-804b-1e25408d8844"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:19:18.419352 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.419260 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-xks-ca" pod "fd24214a-dc9a-4a7b-804b-1e25408d8844" (UID: "fd24214a-dc9a-4a7b-804b-1e25408d8844"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:19:18.419352 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.419305 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd24214a-dc9a-4a7b-804b-1e25408d8844-kube-api-access-cggn6" pod "fd24214a-dc9a-4a7b-804b-1e25408d8844" (UID: "fd24214a-dc9a-4a7b-804b-1e25408d8844"). InnerVolumeSpecName "kube-api-access-cggn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:19:18.519037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.518994 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cggn6\" (UniqueName: \"kubernetes.io/projected/fd24214a-dc9a-4a7b-804b-1e25408d8844-kube-api-access-cggn6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:18.519037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.519027 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:18.519037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:18.519038 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd24214a-dc9a-4a7b-804b-1e25408d8844-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:19.176034 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:19.175992 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" event={"ID":"fd24214a-dc9a-4a7b-804b-1e25408d8844","Type":"ContainerDied","Data":"0e60239a1cedb51105858026ac2defa8b31f87edb1ff44a5a5980620eea4b46d"} Nov 01 21:19:19.176034 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:19.176024 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67" Nov 01 21:19:19.176283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:19.176055 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e60239a1cedb51105858026ac2defa8b31f87edb1ff44a5a5980620eea4b46d" Nov 01 21:19:19.436165 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:19.436075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:19.599696 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:19.599651 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:26.046688 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:26.046655 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn"] Nov 01 21:19:26.095616 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:26.095581 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09e076c6-v2jhn"] Nov 01 21:19:26.466192 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:26.466155 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edda2ac-5594-45c2-b9b4-ec86240b82cc" path="/var/lib/kubelet/pods/5edda2ac-5594-45c2-b9b4-ec86240b82cc/volumes" Nov 01 21:19:27.220642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.220592 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd"] Nov 01 21:19:27.221151 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.221133 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fd24214a-dc9a-4a7b-804b-1e25408d8844" containerName="revoke-keys" Nov 01 21:19:27.221229 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.221154 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd24214a-dc9a-4a7b-804b-1e25408d8844" containerName="revoke-keys" Nov 01 21:19:27.221229 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.221225 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd24214a-dc9a-4a7b-804b-1e25408d8844" containerName="revoke-keys" Nov 01 21:19:27.224632 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.224612 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.280043 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.280006 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd"] Nov 01 21:19:27.315396 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.315358 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bx5\" (UniqueName: \"kubernetes.io/projected/e1ba9b11-d12b-438d-80f8-562f1d761def-kube-api-access-b8bx5\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.315571 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.315509 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.315627 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.315574 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.324890 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.324865 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:19:27.325048 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.324990 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:19:27.325048 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.324995 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:19:27.326299 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.326283 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:19:27.417249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.417211 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bx5\" (UniqueName: \"kubernetes.io/projected/e1ba9b11-d12b-438d-80f8-562f1d761def-kube-api-access-b8bx5\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.417469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.417307 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.417469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.417395 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.417989 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.417968 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.419938 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.419914 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.473414 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.473326 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bx5\" (UniqueName: \"kubernetes.io/projected/e1ba9b11-d12b-438d-80f8-562f1d761def-kube-api-access-b8bx5\") pod \"maas-api-revoke-sub-6b19338f-ch4qd\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.538503 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.538470 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:27.914027 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:27.914000 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd"] Nov 01 21:19:27.914960 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:19:27.914936 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ba9b11_d12b_438d_80f8_562f1d761def.slice/crio-2726612c61c2ce049645995c0f111f47fc7e7afc3145e8a9fce30ac5ec9a9752 WatchSource:0}: Error finding container 2726612c61c2ce049645995c0f111f47fc7e7afc3145e8a9fce30ac5ec9a9752: Status 404 returned error can't find the container with id 2726612c61c2ce049645995c0f111f47fc7e7afc3145e8a9fce30ac5ec9a9752 Nov 01 21:19:28.241450 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:28.241412 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" event={"ID":"e1ba9b11-d12b-438d-80f8-562f1d761def","Type":"ContainerStarted","Data":"091d6b92cf2f000c110aea833160ab3741f3b1a3dc699f36898d67243f14e569"} Nov 01 21:19:28.241844 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:28.241493 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" event={"ID":"e1ba9b11-d12b-438d-80f8-562f1d761def","Type":"ContainerStarted","Data":"2726612c61c2ce049645995c0f111f47fc7e7afc3145e8a9fce30ac5ec9a9752"} Nov 01 21:19:29.248447 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:29.248410 5633 generic.go:356] "Generic (PLEG): container finished" podID="e1ba9b11-d12b-438d-80f8-562f1d761def" containerID="091d6b92cf2f000c110aea833160ab3741f3b1a3dc699f36898d67243f14e569" exitCode=0 Nov 01 21:19:29.248881 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:29.248494 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" event={"ID":"e1ba9b11-d12b-438d-80f8-562f1d761def","Type":"ContainerDied","Data":"091d6b92cf2f000c110aea833160ab3741f3b1a3dc699f36898d67243f14e569"} Nov 01 21:19:30.404409 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.404378 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:30.553093 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.552989 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-openshift-ca\") pod \"e1ba9b11-d12b-438d-80f8-562f1d761def\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " Nov 01 21:19:30.553287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.553113 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-xks-ca\") pod \"e1ba9b11-d12b-438d-80f8-562f1d761def\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " Nov 01 21:19:30.553287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.553169 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ba9b11-d12b-438d-80f8-562f1d761def-kube-api-access-b8bx5\" (UniqueName: \"kubernetes.io/projected/e1ba9b11-d12b-438d-80f8-562f1d761def-kube-api-access-b8bx5\") pod \"e1ba9b11-d12b-438d-80f8-562f1d761def\" (UID: \"e1ba9b11-d12b-438d-80f8-562f1d761def\") " Nov 01 21:19:30.553436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.553382 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-openshift-ca" pod "e1ba9b11-d12b-438d-80f8-562f1d761def" (UID: "e1ba9b11-d12b-438d-80f8-562f1d761def"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:19:30.553672 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.553654 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:30.555414 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.555391 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-xks-ca" pod "e1ba9b11-d12b-438d-80f8-562f1d761def" (UID: "e1ba9b11-d12b-438d-80f8-562f1d761def"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:19:30.555539 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.555519 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ba9b11-d12b-438d-80f8-562f1d761def-kube-api-access-b8bx5" pod "e1ba9b11-d12b-438d-80f8-562f1d761def" (UID: "e1ba9b11-d12b-438d-80f8-562f1d761def"). InnerVolumeSpecName "kube-api-access-b8bx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:19:30.655102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.655043 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ba9b11-d12b-438d-80f8-562f1d761def-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:30.655102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:30.655092 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8bx5\" (UniqueName: \"kubernetes.io/projected/e1ba9b11-d12b-438d-80f8-562f1d761def-kube-api-access-b8bx5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:19:31.261975 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:31.261944 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" Nov 01 21:19:31.261975 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:31.261963 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd" event={"ID":"e1ba9b11-d12b-438d-80f8-562f1d761def","Type":"ContainerDied","Data":"2726612c61c2ce049645995c0f111f47fc7e7afc3145e8a9fce30ac5ec9a9752"} Nov 01 21:19:31.262216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:31.262009 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2726612c61c2ce049645995c0f111f47fc7e7afc3145e8a9fce30ac5ec9a9752" Nov 01 21:19:39.044768 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:39.044730 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp"] Nov 01 21:19:39.094159 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:39.094120 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c295f91-k8sgp"] Nov 01 21:19:39.982062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:39.982029 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:19:40.467290 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:40.467255 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e65d63a-813b-4c45-95b3-08f1fbf3f4c8" path="/var/lib/kubelet/pods/0e65d63a-813b-4c45-95b3-08f1fbf3f4c8/volumes" Nov 01 21:19:41.432199 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:41.432159 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:43.493516 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:43.493475 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:43.741896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:43.741857 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:44.115306 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.115267 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp"] Nov 01 21:19:44.115795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.115782 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e1ba9b11-d12b-438d-80f8-562f1d761def" containerName="revoke-keys" Nov 01 21:19:44.115847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.115797 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ba9b11-d12b-438d-80f8-562f1d761def" containerName="revoke-keys" Nov 01 21:19:44.115899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.115889 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ba9b11-d12b-438d-80f8-562f1d761def" containerName="revoke-keys" Nov 01 21:19:44.120297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.120279 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:44.175226 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.175190 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp"] Nov 01 21:19:44.208899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.208862 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:44.219899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.219870 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-78927386-epp-sa-dockercfg-tkmgv\"" Nov 01 21:19:44.220059 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.219945 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-78927386-kserve-self-signed-certs\"" Nov 01 21:19:44.297069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.297030 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0c6921-4827-4a22-8ca0-d69b913fd086-tls-certs\") pod \"e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:44.297069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.297065 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2j55\" (UniqueName: \"kubernetes.io/projected/4f0c6921-4827-4a22-8ca0-d69b913fd086-kube-api-access-j2j55\") pod \"e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:44.398347 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.398236 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0c6921-4827-4a22-8ca0-d69b913fd086-tls-certs\") pod \"e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:44.398347 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.398322 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2j55\" (UniqueName: \"kubernetes.io/projected/4f0c6921-4827-4a22-8ca0-d69b913fd086-kube-api-access-j2j55\") pod \"e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:44.400949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.400923 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0c6921-4827-4a22-8ca0-d69b913fd086-tls-certs\") pod \"e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:44.459278 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.459239 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2j55\" (UniqueName: \"kubernetes.io/projected/4f0c6921-4827-4a22-8ca0-d69b913fd086-kube-api-access-j2j55\") pod \"e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:44.734510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:44.734465 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:45.107681 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:45.107633 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp"] Nov 01 21:19:45.111085 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:19:45.111054 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0c6921_4827_4a22_8ca0_d69b913fd086.slice/crio-4ac5470bb531e8af9a7ff0b2d6c9def62bbb551ecb41c8b6378dd0c53e47fec9 WatchSource:0}: Error finding container 4ac5470bb531e8af9a7ff0b2d6c9def62bbb551ecb41c8b6378dd0c53e47fec9: Status 404 returned error can't find the container with id 4ac5470bb531e8af9a7ff0b2d6c9def62bbb551ecb41c8b6378dd0c53e47fec9 Nov 01 21:19:45.368393 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:45.368280 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" event={"ID":"4f0c6921-4827-4a22-8ca0-d69b913fd086","Type":"ContainerStarted","Data":"4ac5470bb531e8af9a7ff0b2d6c9def62bbb551ecb41c8b6378dd0c53e47fec9"} Nov 01 21:19:46.155930 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:19:46.155872 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:19:47.363615 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:47.363522 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:19:47.382504 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:47.382464 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" event={"ID":"4f0c6921-4827-4a22-8ca0-d69b913fd086","Type":"ContainerStarted","Data":"f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b"} Nov 01 21:19:47.382715 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:47.382546 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:19:47.494709 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:47.494654 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" podStartSLOduration=1.6042910350000001 podStartE2EDuration="3.494639779s" totalImagesPullingTime="1.890348744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:19:47.493658712 +0000 UTC m=+4595.167917969" watchObservedRunningTime="2026-11-01 21:19:47.494639779 +0000 UTC m=+4595.168899034" Nov 01 21:19:49.785048 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:49.785009 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:19:51.058915 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:51.058827 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t"] Nov 01 21:19:51.106508 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:51.106472 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10c03ca-pxc6t"] Nov 01 21:19:52.469695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:52.469665 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638a104a-f1f9-41ac-891e-9ccad23d2540" path="/var/lib/kubelet/pods/638a104a-f1f9-41ac-891e-9ccad23d2540/volumes" Nov 01 21:19:55.280455 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:19:55.280413 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:20:16.156572 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:20:16.156533 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:20:18.390716 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:20:18.390686 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:20:18.474984 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:20:18.474951 5633 scope.go:122] "RemoveContainer" containerID="546a191410ae8ed9444ad211e827ecdf55d5dd53566cad09078d73fea0328f98" Nov 01 21:20:18.484647 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:20:18.484628 5633 scope.go:122] "RemoveContainer" containerID="d44443e3550da26b5d5a4959c637fb339ca6fe45d7a10154f8720a99fbf3145e" Nov 01 21:20:18.493375 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:20:18.493345 5633 scope.go:122] "RemoveContainer" containerID="1b35361b344094e89ef66f65d57ff1c40cb7cff1d15e5ca62149a98a8fbad06c" Nov 01 21:20:30.860702 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:20:30.860665 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:20:44.068509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:20:44.068464 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:20:46.156487 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:20:46.156448 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:21:01.542744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:01.542710 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp"] Nov 01 21:21:01.589037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:01.588966 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:01.755843 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:01.755808 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:02.188487 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:02.188451 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:21:08.391155 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:08.391096 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:21:11.953868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:11.953832 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:21:16.156806 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:21:16.156768 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:21:16.564221 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:16.564175 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" containerID="cri-o://f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b" gracePeriod=45 Nov 01 21:21:18.390356 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:18.390288 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:21:18.531126 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:18.531093 5633 scope.go:122] "RemoveContainer" containerID="7989d3e723a03cca7e5eab1c48343a7f4ad3b40760e61030eb45d4a7415ff698" Nov 01 21:21:18.543490 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:18.543466 5633 scope.go:122] "RemoveContainer" containerID="2c54ab6d50a546235a77a1b11a91da1ae0536a7e550ddeed130b07773144cefd" Nov 01 21:21:18.552503 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:18.552482 5633 scope.go:122] "RemoveContainer" containerID="a68f8101eac9784fd567551b6c3da0c8ab507ffe9010982d6071da8476c463b3" Nov 01 21:21:19.281372 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:19.281309 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:19.840893 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:19.840846 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:20.209748 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:20.209645 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:23.549412 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:23.548433 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:23.723710 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:23.723630 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:23.855420 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:23.855286 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:26.776198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:26.776160 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:26.951432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:26.951391 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:28.390735 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:28.390699 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:21:28.391157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:28.390802 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:21:29.955119 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:29.955083 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:30.113007 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:30.112974 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:30.286003 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:30.285961 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:34.775955 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:34.775912 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:37.789238 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:37.789186 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:21:38.390810 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:38.390773 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:21:39.505962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:39.505924 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:44.175899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:44.175863 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:44.402449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:44.402414 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:46.156164 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:21:46.156117 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:21:46.824482 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:46.824454 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:21:47.001958 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.001910 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0c6921-4827-4a22-8ca0-d69b913fd086-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0c6921-4827-4a22-8ca0-d69b913fd086-tls-certs\") pod \"4f0c6921-4827-4a22-8ca0-d69b913fd086\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " Nov 01 21:21:47.002176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.002014 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0c6921-4827-4a22-8ca0-d69b913fd086-kube-api-access-j2j55\" (UniqueName: \"kubernetes.io/projected/4f0c6921-4827-4a22-8ca0-d69b913fd086-kube-api-access-j2j55\") pod \"4f0c6921-4827-4a22-8ca0-d69b913fd086\" (UID: \"4f0c6921-4827-4a22-8ca0-d69b913fd086\") " Nov 01 21:21:47.004699 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.004656 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0c6921-4827-4a22-8ca0-d69b913fd086-kube-api-access-j2j55" pod "4f0c6921-4827-4a22-8ca0-d69b913fd086" (UID: "4f0c6921-4827-4a22-8ca0-d69b913fd086"). InnerVolumeSpecName "kube-api-access-j2j55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:21:47.004699 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.004662 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0c6921-4827-4a22-8ca0-d69b913fd086-tls-certs" pod "4f0c6921-4827-4a22-8ca0-d69b913fd086" (UID: "4f0c6921-4827-4a22-8ca0-d69b913fd086"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:21:47.103950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.103916 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2j55\" (UniqueName: \"kubernetes.io/projected/4f0c6921-4827-4a22-8ca0-d69b913fd086-kube-api-access-j2j55\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:47.103950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.103944 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0c6921-4827-4a22-8ca0-d69b913fd086-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:21:47.221888 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.221851 5633 generic.go:356] "Generic (PLEG): container finished" podID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerID="f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b" exitCode=0 Nov 01 21:21:47.222313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.221896 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" Nov 01 21:21:47.222313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.221937 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" event={"ID":"4f0c6921-4827-4a22-8ca0-d69b913fd086","Type":"ContainerDied","Data":"f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b"} Nov 01 21:21:47.222313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.221994 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp" event={"ID":"4f0c6921-4827-4a22-8ca0-d69b913fd086","Type":"ContainerDied","Data":"4ac5470bb531e8af9a7ff0b2d6c9def62bbb551ecb41c8b6378dd0c53e47fec9"} Nov 01 21:21:47.222313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.222022 5633 scope.go:122] "RemoveContainer" containerID="f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b" Nov 01 21:21:47.232307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.232281 5633 scope.go:122] "RemoveContainer" containerID="f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b" Nov 01 21:21:47.232655 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:21:47.232625 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b\": container with ID starting with f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b not found: ID does not exist" containerID="f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b" Nov 01 21:21:47.232725 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.232665 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b"} err="failed to get container status \"f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b\": rpc error: code = NotFound desc = could not find container \"f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b\": container with ID starting with f66565087984b5eac84fbe2127021da1d4414ceda08d1ea6761f86da9027a85b not found: ID does not exist" Nov 01 21:21:47.383875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.383774 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp"] Nov 01 21:21:47.387666 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:47.387636 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-router-scheduler-6c8fcd7dc4-7tfcp"] Nov 01 21:21:48.320909 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:48.320875 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:48.465801 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:48.465764 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" path="/var/lib/kubelet/pods/4f0c6921-4827-4a22-8ca0-d69b913fd086/volumes" Nov 01 21:21:48.879449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:48.879414 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:49.069870 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:49.069838 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:49.196852 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:49.196767 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:51.629444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.629396 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn"] Nov 01 21:21:51.630102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.630079 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" Nov 01 21:21:51.630207 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.630105 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" Nov 01 21:21:51.630283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.630266 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0c6921-4827-4a22-8ca0-d69b913fd086" containerName="main" Nov 01 21:21:51.633489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.633465 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.658917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.658881 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a930675-2403-4ede-934f-8fb81476d360-tls-certs\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.659134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.659052 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-dshm\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.659208 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.659192 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6c2\" (UniqueName: \"kubernetes.io/projected/5a930675-2403-4ede-934f-8fb81476d360-kube-api-access-5z6c2\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.659269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.659243 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-model-cache\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.659436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.659413 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-tmp-dir\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.659560 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.659462 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-home\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.704485 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.704449 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn"] Nov 01 21:21:51.704824 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.704799 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:51.734665 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.734627 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f3eeb7\"/\"kube-root-ca.crt\"" Nov 01 21:21:51.734869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.734682 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f3eeb7\"/\"openshift-service-ca.crt\"" Nov 01 21:21:51.734869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.734732 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f3eeb7\"/\"default-dockercfg-shfwv\"" Nov 01 21:21:51.734869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.734744 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f3eeb7\"/\"simulator-single-main-f3eeb7-kserve-self-signed-certs\"" Nov 01 21:21:51.760275 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.760232 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6c2\" (UniqueName: \"kubernetes.io/projected/5a930675-2403-4ede-934f-8fb81476d360-kube-api-access-5z6c2\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.760509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.760292 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-model-cache\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.760509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.760372 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-tmp-dir\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.760509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.760414 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-home\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.760509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.760455 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a930675-2403-4ede-934f-8fb81476d360-tls-certs\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.760803 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.760554 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-dshm\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.760982 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.760896 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-model-cache\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.761090 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.761066 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-tmp-dir\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.761261 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.761237 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-home\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.763751 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.763721 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-dshm\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.764048 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.764027 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a930675-2403-4ede-934f-8fb81476d360-tls-certs\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.821159 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.821117 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6c2\" (UniqueName: \"kubernetes.io/projected/5a930675-2403-4ede-934f-8fb81476d360-kube-api-access-5z6c2\") pod \"simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:51.946318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:51.946215 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:52.324125 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:52.324098 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn"] Nov 01 21:21:52.325461 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:21:52.325427 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a930675_2403_4ede_934f_8fb81476d360.slice/crio-f5211284efa08e67e995ff92a7677a9d49876aefcaa452b4fccf9cff8ae241c1 WatchSource:0}: Error finding container f5211284efa08e67e995ff92a7677a9d49876aefcaa452b4fccf9cff8ae241c1: Status 404 returned error can't find the container with id f5211284efa08e67e995ff92a7677a9d49876aefcaa452b4fccf9cff8ae241c1 Nov 01 21:21:53.262119 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:53.262078 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" event={"ID":"5a930675-2403-4ede-934f-8fb81476d360","Type":"ContainerStarted","Data":"d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94"} Nov 01 21:21:53.262119 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:53.262129 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" event={"ID":"5a930675-2403-4ede-934f-8fb81476d360","Type":"ContainerStarted","Data":"f5211284efa08e67e995ff92a7677a9d49876aefcaa452b4fccf9cff8ae241c1"} Nov 01 21:21:53.375566 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:53.375502 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" podStartSLOduration=2.375487319 podStartE2EDuration="2.375487319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:53.373642148 +0000 UTC m=+4721.047901431" watchObservedRunningTime="2026-11-01 21:21:53.375487319 +0000 UTC m=+4721.049746574" Nov 01 21:21:53.772647 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:53.772591 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:21:53.946782 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:53.946743 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:21:56.194544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:56.194507 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:21:58.545982 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:21:58.545945 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:01.946531 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:01.946501 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:22:01.959069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:01.959034 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:22:02.326209 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:02.326172 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:22:03.180466 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:03.180427 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:07.877492 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:07.877452 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:12.583227 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:12.583192 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:16.157573 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:22:16.157533 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:22:17.278394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:17.278356 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:21.886614 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:21.886576 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:22:21.973052 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:21.973015 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:26.713572 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:26.713540 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:31.387074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:31.387027 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:35.044731 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:35.044693 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf"] Nov 01 21:22:35.094028 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:35.093987 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pfdxf"] Nov 01 21:22:36.077463 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:36.077422 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:36.467308 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:36.467274 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c23098-bf82-4db0-8156-2cb96d4ffeba" path="/var/lib/kubelet/pods/e2c23098-bf82-4db0-8156-2cb96d4ffeba/volumes" Nov 01 21:22:36.947505 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:36.947425 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld"] Nov 01 21:22:36.951449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:36.951424 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.007630 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.007598 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld"] Nov 01 21:22:37.037553 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.037521 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwj4c\" (UniqueName: \"kubernetes.io/projected/4cd1f6b2-2380-4c5a-8f39-30198da8f197-kube-api-access-dwj4c\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.037762 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.037561 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.037762 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.037601 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.052515 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.052485 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:22:37.052704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.052568 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:22:37.052704 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.052609 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:22:37.053078 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.053066 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:22:37.139029 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.138996 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwj4c\" (UniqueName: \"kubernetes.io/projected/4cd1f6b2-2380-4c5a-8f39-30198da8f197-kube-api-access-dwj4c\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.139029 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.139039 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.139650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.139085 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.139789 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.139768 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.141717 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.141690 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.195344 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.195295 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwj4c\" (UniqueName: \"kubernetes.io/projected/4cd1f6b2-2380-4c5a-8f39-30198da8f197-kube-api-access-dwj4c\") pod \"maas-api-revoke-sub-f4c3fe42-5j6ld\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.265266 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.265229 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:37.639835 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:37.639804 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld"] Nov 01 21:22:37.640849 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:22:37.640823 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd1f6b2_2380_4c5a_8f39_30198da8f197.slice/crio-4059af72bfb5ae85e38aaf635c57c5fd77a523335acf82f7cdbd3c3deec7e504 WatchSource:0}: Error finding container 4059af72bfb5ae85e38aaf635c57c5fd77a523335acf82f7cdbd3c3deec7e504: Status 404 returned error can't find the container with id 4059af72bfb5ae85e38aaf635c57c5fd77a523335acf82f7cdbd3c3deec7e504 Nov 01 21:22:38.524247 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:38.524208 5633 generic.go:356] "Generic (PLEG): container finished" podID="4cd1f6b2-2380-4c5a-8f39-30198da8f197" containerID="2f33584c275a781f4d5bfe1579cbe8434a9f11e088b337d54fb252631fbf7dd0" exitCode=0 Nov 01 21:22:38.524247 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:38.524243 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" event={"ID":"4cd1f6b2-2380-4c5a-8f39-30198da8f197","Type":"ContainerDied","Data":"2f33584c275a781f4d5bfe1579cbe8434a9f11e088b337d54fb252631fbf7dd0"} Nov 01 21:22:38.524716 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:38.524295 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" event={"ID":"4cd1f6b2-2380-4c5a-8f39-30198da8f197","Type":"ContainerStarted","Data":"4059af72bfb5ae85e38aaf635c57c5fd77a523335acf82f7cdbd3c3deec7e504"} Nov 01 21:22:39.686283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.686250 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:39.769692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.769650 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-xks-ca\") pod \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " Nov 01 21:22:39.769900 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.769727 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd1f6b2-2380-4c5a-8f39-30198da8f197-kube-api-access-dwj4c\" (UniqueName: \"kubernetes.io/projected/4cd1f6b2-2380-4c5a-8f39-30198da8f197-kube-api-access-dwj4c\") pod \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " Nov 01 21:22:39.769900 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.769778 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-openshift-ca\") pod \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\" (UID: \"4cd1f6b2-2380-4c5a-8f39-30198da8f197\") " Nov 01 21:22:39.770154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.770127 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-openshift-ca" pod "4cd1f6b2-2380-4c5a-8f39-30198da8f197" (UID: "4cd1f6b2-2380-4c5a-8f39-30198da8f197"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:22:39.772023 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.771995 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-xks-ca" pod "4cd1f6b2-2380-4c5a-8f39-30198da8f197" (UID: "4cd1f6b2-2380-4c5a-8f39-30198da8f197"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:22:39.772122 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.772034 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd1f6b2-2380-4c5a-8f39-30198da8f197-kube-api-access-dwj4c" pod "4cd1f6b2-2380-4c5a-8f39-30198da8f197" (UID: "4cd1f6b2-2380-4c5a-8f39-30198da8f197"). InnerVolumeSpecName "kube-api-access-dwj4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:22:39.871108 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.870941 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:39.871108 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.870990 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwj4c\" (UniqueName: \"kubernetes.io/projected/4cd1f6b2-2380-4c5a-8f39-30198da8f197-kube-api-access-dwj4c\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:39.871108 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:39.871007 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1f6b2-2380-4c5a-8f39-30198da8f197-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:22:40.537778 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:40.537747 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" Nov 01 21:22:40.537969 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:40.537782 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld" event={"ID":"4cd1f6b2-2380-4c5a-8f39-30198da8f197","Type":"ContainerDied","Data":"4059af72bfb5ae85e38aaf635c57c5fd77a523335acf82f7cdbd3c3deec7e504"} Nov 01 21:22:40.537969 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:40.537843 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4059af72bfb5ae85e38aaf635c57c5fd77a523335acf82f7cdbd3c3deec7e504" Nov 01 21:22:40.762540 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:40.762442 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:44.753039 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:44.753000 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:46.049138 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:46.049098 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:46.157131 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:22:46.157097 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:22:48.019632 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:48.019592 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:48.473372 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:48.473320 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:50.009477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:50.009444 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:22:50.095649 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:50.095617 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:52.989042 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:52.988945 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn"] Nov 01 21:22:53.084668 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:53.084623 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:54.151014 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:54.150979 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:54.320993 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:54.320949 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:54.507884 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:54.507836 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:54.633645 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:54.633608 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:59.656786 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:59.656749 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:22:59.811618 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:22:59.811567 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:02.698768 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:02.698731 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:23:08.010730 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.010684 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" podUID="5a930675-2403-4ede-934f-8fb81476d360" containerName="main" containerID="cri-o://d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94" gracePeriod=45 Nov 01 21:23:08.257556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.257526 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:23:08.370316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370226 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a930675-2403-4ede-934f-8fb81476d360-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a930675-2403-4ede-934f-8fb81476d360-tls-certs\") pod \"5a930675-2403-4ede-934f-8fb81476d360\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " Nov 01 21:23:08.370316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370273 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-home\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-home\") pod \"5a930675-2403-4ede-934f-8fb81476d360\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " Nov 01 21:23:08.371190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370384 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a930675-2403-4ede-934f-8fb81476d360-kube-api-access-5z6c2\" (UniqueName: \"kubernetes.io/projected/5a930675-2403-4ede-934f-8fb81476d360-kube-api-access-5z6c2\") pod \"5a930675-2403-4ede-934f-8fb81476d360\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " Nov 01 21:23:08.371190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370421 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-tmp-dir\") pod \"5a930675-2403-4ede-934f-8fb81476d360\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " Nov 01 21:23:08.371190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370528 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-model-cache\") pod \"5a930675-2403-4ede-934f-8fb81476d360\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " Nov 01 21:23:08.371190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370568 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-dshm\") pod \"5a930675-2403-4ede-934f-8fb81476d360\" (UID: \"5a930675-2403-4ede-934f-8fb81476d360\") " Nov 01 21:23:08.371190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370611 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-home" pod "5a930675-2403-4ede-934f-8fb81476d360" (UID: "5a930675-2403-4ede-934f-8fb81476d360"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:23:08.371190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.370962 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:08.371577 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.371441 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-tmp-dir" pod "5a930675-2403-4ede-934f-8fb81476d360" (UID: "5a930675-2403-4ede-934f-8fb81476d360"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:23:08.371934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.371908 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-model-cache" pod "5a930675-2403-4ede-934f-8fb81476d360" (UID: "5a930675-2403-4ede-934f-8fb81476d360"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:23:08.373416 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.373379 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a930675-2403-4ede-934f-8fb81476d360-kube-api-access-5z6c2" pod "5a930675-2403-4ede-934f-8fb81476d360" (UID: "5a930675-2403-4ede-934f-8fb81476d360"). InnerVolumeSpecName "kube-api-access-5z6c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:23:08.373621 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.373586 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-dshm" pod "5a930675-2403-4ede-934f-8fb81476d360" (UID: "5a930675-2403-4ede-934f-8fb81476d360"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:23:08.374721 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.374671 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a930675-2403-4ede-934f-8fb81476d360-tls-certs" pod "5a930675-2403-4ede-934f-8fb81476d360" (UID: "5a930675-2403-4ede-934f-8fb81476d360"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:23:08.471793 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.471756 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6c2\" (UniqueName: \"kubernetes.io/projected/5a930675-2403-4ede-934f-8fb81476d360-kube-api-access-5z6c2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:08.471793 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.471795 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:08.472021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.471811 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:08.472021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.471826 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a930675-2403-4ede-934f-8fb81476d360-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:08.472021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.471842 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a930675-2403-4ede-934f-8fb81476d360-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:08.700997 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.700895 5633 generic.go:356] "Generic (PLEG): container finished" podID="5a930675-2403-4ede-934f-8fb81476d360" containerID="d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94" exitCode=0 Nov 01 21:23:08.700997 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.700943 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" Nov 01 21:23:08.700997 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.700978 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" event={"ID":"5a930675-2403-4ede-934f-8fb81476d360","Type":"ContainerDied","Data":"d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94"} Nov 01 21:23:08.701290 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.701033 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn" event={"ID":"5a930675-2403-4ede-934f-8fb81476d360","Type":"ContainerDied","Data":"f5211284efa08e67e995ff92a7677a9d49876aefcaa452b4fccf9cff8ae241c1"} Nov 01 21:23:08.701290 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.701056 5633 scope.go:122] "RemoveContainer" containerID="d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94" Nov 01 21:23:08.710687 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.710664 5633 scope.go:122] "RemoveContainer" containerID="d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94" Nov 01 21:23:08.710990 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:23:08.710970 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94\": container with ID starting with d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94 not found: ID does not exist" containerID="d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94" Nov 01 21:23:08.711051 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.711002 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94"} err="failed to get container status \"d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94\": rpc error: code = NotFound desc = could not find container \"d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94\": container with ID starting with d64ec28da5e30754b64b9a4f196d1e8b4bc54485763ffc6ce691cd06c9634e94 not found: ID does not exist" Nov 01 21:23:08.863353 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.863296 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn"] Nov 01 21:23:08.912310 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:08.912276 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f3eeb7/simulator-single-main-f3eeb7-kserve-567c6f967b-qhtxn"] Nov 01 21:23:10.466708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:10.466671 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a930675-2403-4ede-934f-8fb81476d360" path="/var/lib/kubelet/pods/5a930675-2403-4ede-934f-8fb81476d360/volumes" Nov 01 21:23:15.637153 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:15.637112 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:15.753008 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:15.752971 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:16.155946 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:23:16.155910 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:23:18.600473 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:18.600440 5633 scope.go:122] "RemoveContainer" containerID="8f7777d2595c966829b779fddb1350a33759a532bd493c9b913e719e0f424ee7" Nov 01 21:23:28.606638 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:28.606596 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:28.779698 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:28.779655 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:29.202402 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:29.202368 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:30.346640 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:30.346602 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:30.833589 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:30.833554 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:30.995889 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:30.995849 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:31.117066 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:31.116974 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:31.718007 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:31.717975 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:31.949779 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:31.949742 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:35.704231 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:35.704196 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:36.034074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:36.034008 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:36.157968 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:36.157879 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:40.377394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:40.377355 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:43.521184 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.521148 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn"] Nov 01 21:23:43.521686 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.521670 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5a930675-2403-4ede-934f-8fb81476d360" containerName="main" Nov 01 21:23:43.521733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.521688 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a930675-2403-4ede-934f-8fb81476d360" containerName="main" Nov 01 21:23:43.521733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.521726 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4cd1f6b2-2380-4c5a-8f39-30198da8f197" containerName="revoke-keys" Nov 01 21:23:43.521733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.521732 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd1f6b2-2380-4c5a-8f39-30198da8f197" containerName="revoke-keys" Nov 01 21:23:43.521887 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.521812 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd1f6b2-2380-4c5a-8f39-30198da8f197" containerName="revoke-keys" Nov 01 21:23:43.521887 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.521822 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a930675-2403-4ede-934f-8fb81476d360" containerName="main" Nov 01 21:23:43.525273 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.525254 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.565779 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.565743 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:23:43.580367 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.580317 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn"] Nov 01 21:23:43.611961 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.611927 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:23:43.611961 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.611961 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:23:43.612202 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.611975 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:23:43.612202 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.612017 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:23:43.634825 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.634779 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vjj\" (UniqueName: \"kubernetes.io/projected/589bab5b-6316-477a-82f9-1a21e05c9e25-kube-api-access-w7vjj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.635023 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.634989 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.635202 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.635178 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.736591 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.736552 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vjj\" (UniqueName: \"kubernetes.io/projected/589bab5b-6316-477a-82f9-1a21e05c9e25-kube-api-access-w7vjj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.736816 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.736642 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.736816 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.736737 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.737431 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.737411 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.739340 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.739305 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.793480 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.793405 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vjj\" (UniqueName: \"kubernetes.io/projected/589bab5b-6316-477a-82f9-1a21e05c9e25-kube-api-access-w7vjj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:43.838364 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:43.838309 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:44.212134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:44.212108 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn"] Nov 01 21:23:44.213998 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:23:44.213965 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589bab5b_6316_477a_82f9_1a21e05c9e25.slice/crio-c78ed812e103226c11f8540c5f705dbb68ab39e37c3a1e15a819e56db0dbb634 WatchSource:0}: Error finding container c78ed812e103226c11f8540c5f705dbb68ab39e37c3a1e15a819e56db0dbb634: Status 404 returned error can't find the container with id c78ed812e103226c11f8540c5f705dbb68ab39e37c3a1e15a819e56db0dbb634 Nov 01 21:23:44.927103 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:44.926998 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" event={"ID":"589bab5b-6316-477a-82f9-1a21e05c9e25","Type":"ContainerStarted","Data":"ba53f1cd6fa411ac94b1da063cf84a7633fedb1c74c12e8edae6ba303f6c727e"} Nov 01 21:23:44.927103 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:44.927061 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" event={"ID":"589bab5b-6316-477a-82f9-1a21e05c9e25","Type":"ContainerStarted","Data":"c78ed812e103226c11f8540c5f705dbb68ab39e37c3a1e15a819e56db0dbb634"} Nov 01 21:23:45.085557 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:45.085497 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" podStartSLOduration=2.085482835 podStartE2EDuration="2.085482835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:23:45.038120036 +0000 UTC m=+4832.712379290" watchObservedRunningTime="2026-11-01 21:23:45.085482835 +0000 UTC m=+4832.759742090" Nov 01 21:23:45.132311 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:45.132275 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:45.250621 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:45.250589 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:45.934213 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:45.934178 5633 generic.go:356] "Generic (PLEG): container finished" podID="589bab5b-6316-477a-82f9-1a21e05c9e25" containerID="ba53f1cd6fa411ac94b1da063cf84a7633fedb1c74c12e8edae6ba303f6c727e" exitCode=0 Nov 01 21:23:45.934724 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:45.934278 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" event={"ID":"589bab5b-6316-477a-82f9-1a21e05c9e25","Type":"ContainerDied","Data":"ba53f1cd6fa411ac94b1da063cf84a7633fedb1c74c12e8edae6ba303f6c727e"} Nov 01 21:23:46.156290 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:23:46.156252 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:23:47.087108 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.087074 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:47.174822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.174782 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-openshift-ca\") pod \"589bab5b-6316-477a-82f9-1a21e05c9e25\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " Nov 01 21:23:47.174822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.174821 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589bab5b-6316-477a-82f9-1a21e05c9e25-kube-api-access-w7vjj\" (UniqueName: \"kubernetes.io/projected/589bab5b-6316-477a-82f9-1a21e05c9e25-kube-api-access-w7vjj\") pod \"589bab5b-6316-477a-82f9-1a21e05c9e25\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " Nov 01 21:23:47.175084 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.174960 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-xks-ca\") pod \"589bab5b-6316-477a-82f9-1a21e05c9e25\" (UID: \"589bab5b-6316-477a-82f9-1a21e05c9e25\") " Nov 01 21:23:47.175182 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.175161 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-openshift-ca" pod "589bab5b-6316-477a-82f9-1a21e05c9e25" (UID: "589bab5b-6316-477a-82f9-1a21e05c9e25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:23:47.175462 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.175443 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:47.177091 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.177070 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-xks-ca" pod "589bab5b-6316-477a-82f9-1a21e05c9e25" (UID: "589bab5b-6316-477a-82f9-1a21e05c9e25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:23:47.177209 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.177190 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589bab5b-6316-477a-82f9-1a21e05c9e25-kube-api-access-w7vjj" pod "589bab5b-6316-477a-82f9-1a21e05c9e25" (UID: "589bab5b-6316-477a-82f9-1a21e05c9e25"). InnerVolumeSpecName "kube-api-access-w7vjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:23:47.276986 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.276945 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vjj\" (UniqueName: \"kubernetes.io/projected/589bab5b-6316-477a-82f9-1a21e05c9e25-kube-api-access-w7vjj\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:47.276986 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.276978 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589bab5b-6316-477a-82f9-1a21e05c9e25-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:23:47.946518 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.946481 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" Nov 01 21:23:47.946518 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.946493 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn" event={"ID":"589bab5b-6316-477a-82f9-1a21e05c9e25","Type":"ContainerDied","Data":"c78ed812e103226c11f8540c5f705dbb68ab39e37c3a1e15a819e56db0dbb634"} Nov 01 21:23:47.946753 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:47.946544 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78ed812e103226c11f8540c5f705dbb68ab39e37c3a1e15a819e56db0dbb634" Nov 01 21:23:49.782974 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:49.782934 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:54.403005 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:54.402957 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:56.156454 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:56.156405 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:56.332012 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:56.331958 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:58.499748 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:58.499720 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:23:59.043398 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:59.043363 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg"] Nov 01 21:23:59.092557 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:23:59.092523 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mn9dg"] Nov 01 21:24:00.010360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:00.010309 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:24:00.287297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:00.287182 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:24:00.466999 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:00.466965 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db1c6d9-1e9c-4d45-855f-61de797cffa1" path="/var/lib/kubelet/pods/5db1c6d9-1e9c-4d45-855f-61de797cffa1/volumes" Nov 01 21:24:00.615672 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:00.615582 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn"] Nov 01 21:24:00.665193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:00.665157 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f3eeb7-kbrpn"] Nov 01 21:24:00.881316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:00.881214 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:24:02.471345 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:02.471283 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589bab5b-6316-477a-82f9-1a21e05c9e25" path="/var/lib/kubelet/pods/589bab5b-6316-477a-82f9-1a21e05c9e25/volumes" Nov 01 21:24:04.685915 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:04.685873 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:24:16.156399 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:24:16.156360 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:24:17.220306 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.220265 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r"] Nov 01 21:24:17.220780 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.220766 5633 cpu_manager.go:414] "containerMap: removing container" podUID="589bab5b-6316-477a-82f9-1a21e05c9e25" containerName="revoke-keys" Nov 01 21:24:17.220827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.220781 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589bab5b-6316-477a-82f9-1a21e05c9e25" containerName="revoke-keys" Nov 01 21:24:17.220879 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.220870 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="589bab5b-6316-477a-82f9-1a21e05c9e25" containerName="revoke-keys" Nov 01 21:24:17.224020 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.224002 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.295102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.295065 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d857cd33-2469-441a-9ac0-d1c6b27ba948-tls-certs\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.295358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.295130 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-model-cache\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.295358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.295257 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-home\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.295358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.295309 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bfz\" (UniqueName: \"kubernetes.io/projected/d857cd33-2469-441a-9ac0-d1c6b27ba948-kube-api-access-b4bfz\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.295518 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.295443 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-dshm\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.295518 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.295501 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-tmp-dir\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.311397 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.311358 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:24:17.326173 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.326138 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9cd91242-kserve-self-signed-certs\"" Nov 01 21:24:17.329086 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.329060 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r"] Nov 01 21:24:17.396591 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.396551 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bfz\" (UniqueName: \"kubernetes.io/projected/d857cd33-2469-441a-9ac0-d1c6b27ba948-kube-api-access-b4bfz\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.396818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.396605 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-dshm\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.396818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.396646 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-tmp-dir\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.396818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.396718 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d857cd33-2469-441a-9ac0-d1c6b27ba948-tls-certs\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.396818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.396787 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-model-cache\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.397089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.396847 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-home\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.397195 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.397157 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-tmp-dir\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.397319 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.397213 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-home\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.397319 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.397226 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-model-cache\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.399129 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.399106 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-dshm\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.399524 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.399505 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d857cd33-2469-441a-9ac0-d1c6b27ba948-tls-certs\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.453102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.453059 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bfz\" (UniqueName: \"kubernetes.io/projected/d857cd33-2469-441a-9ac0-d1c6b27ba948-kube-api-access-b4bfz\") pod \"e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.536249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.536152 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:17.918272 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:17.918240 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r"] Nov 01 21:24:17.919365 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:24:17.919320 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd857cd33_2469_441a_9ac0_d1c6b27ba948.slice/crio-7e007fc94703e18242c68cfb475f13aa45d7b5846862f59604f4ef7d9c15085c WatchSource:0}: Error finding container 7e007fc94703e18242c68cfb475f13aa45d7b5846862f59604f4ef7d9c15085c: Status 404 returned error can't find the container with id 7e007fc94703e18242c68cfb475f13aa45d7b5846862f59604f4ef7d9c15085c Nov 01 21:24:18.132747 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:18.132709 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" event={"ID":"d857cd33-2469-441a-9ac0-d1c6b27ba948","Type":"ContainerStarted","Data":"c784092370b367e9afe17728851fe7c6216603f9bd0b4fcbe97bfb1dbed639e1"} Nov 01 21:24:18.133034 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:18.132755 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" event={"ID":"d857cd33-2469-441a-9ac0-d1c6b27ba948","Type":"ContainerStarted","Data":"7e007fc94703e18242c68cfb475f13aa45d7b5846862f59604f4ef7d9c15085c"} Nov 01 21:24:18.536481 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:18.536433 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:18.628182 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:18.628136 5633 scope.go:122] "RemoveContainer" containerID="4b12a847ede21fd1ff0ae132bc0d257439ab5a664c166e1b89e05f1635b7b77b" Nov 01 21:24:19.044585 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:19.044534 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" podStartSLOduration=2.044517601 podStartE2EDuration="2.044517601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:24:18.24472594 +0000 UTC m=+4865.918985200" watchObservedRunningTime="2026-11-01 21:24:19.044517601 +0000 UTC m=+4866.718776856" Nov 01 21:24:19.044796 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:19.044746 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67"] Nov 01 21:24:19.093812 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:19.093774 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7wm67"] Nov 01 21:24:20.465795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:20.465764 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd24214a-dc9a-4a7b-804b-1e25408d8844" path="/var/lib/kubelet/pods/fd24214a-dc9a-4a7b-804b-1e25408d8844/volumes" Nov 01 21:24:21.779474 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:21.779389 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:24:27.536745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:27.536717 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:27.549296 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:27.549262 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:28.204917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:28.204884 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:24:31.044743 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:31.044709 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd"] Nov 01 21:24:31.054589 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:31.054559 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ch4qd"] Nov 01 21:24:32.466543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:32.466508 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ba9b11-d12b-438d-80f8-562f1d761def" path="/var/lib/kubelet/pods/e1ba9b11-d12b-438d-80f8-562f1d761def/volumes" Nov 01 21:24:45.341420 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:45.341384 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:24:45.513757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:45.513721 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:24:46.157169 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:24:46.157132 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:24:49.974985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:24:49.974945 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:25:10.008904 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:10.008865 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:25:16.157035 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:25:16.156994 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:25:18.651859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:18.651827 5633 scope.go:122] "RemoveContainer" containerID="3268ff3a970022320d0ad3e98caddd757ee5bb442726c67d8edabcb6a4fd1753" Nov 01 21:25:18.661252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:18.661226 5633 scope.go:122] "RemoveContainer" containerID="bf0a2f3a002e510b2305fd06341e1d565ca434799ecb8bd03892a2e694ca38a7" Nov 01 21:25:21.336326 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:21.336289 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r"] Nov 01 21:25:22.958908 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:22.958873 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:24.080252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:24.080213 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:25:29.761153 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:29.761115 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:36.358031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.357850 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" podUID="d857cd33-2469-441a-9ac0-d1c6b27ba948" containerName="main" containerID="cri-o://c784092370b367e9afe17728851fe7c6216603f9bd0b4fcbe97bfb1dbed639e1" gracePeriod=45 Nov 01 21:25:36.773729 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.773621 5633 generic.go:356] "Generic (PLEG): container finished" podID="d857cd33-2469-441a-9ac0-d1c6b27ba948" containerID="c784092370b367e9afe17728851fe7c6216603f9bd0b4fcbe97bfb1dbed639e1" exitCode=0 Nov 01 21:25:36.773729 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.773675 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" event={"ID":"d857cd33-2469-441a-9ac0-d1c6b27ba948","Type":"ContainerDied","Data":"c784092370b367e9afe17728851fe7c6216603f9bd0b4fcbe97bfb1dbed639e1"} Nov 01 21:25:36.882490 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.882458 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:25:36.977542 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.977507 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d857cd33-2469-441a-9ac0-d1c6b27ba948-tls-certs\" (UniqueName: \"kubernetes.io/secret/d857cd33-2469-441a-9ac0-d1c6b27ba948-tls-certs\") pod \"d857cd33-2469-441a-9ac0-d1c6b27ba948\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " Nov 01 21:25:36.977771 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.977570 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-home\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-home\") pod \"d857cd33-2469-441a-9ac0-d1c6b27ba948\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " Nov 01 21:25:36.977771 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.977637 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d857cd33-2469-441a-9ac0-d1c6b27ba948-kube-api-access-b4bfz\" (UniqueName: \"kubernetes.io/projected/d857cd33-2469-441a-9ac0-d1c6b27ba948-kube-api-access-b4bfz\") pod \"d857cd33-2469-441a-9ac0-d1c6b27ba948\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " Nov 01 21:25:36.977771 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.977722 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-model-cache\") pod \"d857cd33-2469-441a-9ac0-d1c6b27ba948\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " Nov 01 21:25:36.977981 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.977805 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-tmp-dir\") pod \"d857cd33-2469-441a-9ac0-d1c6b27ba948\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " Nov 01 21:25:36.977981 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.977875 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-dshm\") pod \"d857cd33-2469-441a-9ac0-d1c6b27ba948\" (UID: \"d857cd33-2469-441a-9ac0-d1c6b27ba948\") " Nov 01 21:25:36.978894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.978838 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-tmp-dir" pod "d857cd33-2469-441a-9ac0-d1c6b27ba948" (UID: "d857cd33-2469-441a-9ac0-d1c6b27ba948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:36.978894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.978868 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-home" pod "d857cd33-2469-441a-9ac0-d1c6b27ba948" (UID: "d857cd33-2469-441a-9ac0-d1c6b27ba948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:36.979160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.979131 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-model-cache" pod "d857cd33-2469-441a-9ac0-d1c6b27ba948" (UID: "d857cd33-2469-441a-9ac0-d1c6b27ba948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:36.981317 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.981283 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d857cd33-2469-441a-9ac0-d1c6b27ba948-kube-api-access-b4bfz" pod "d857cd33-2469-441a-9ac0-d1c6b27ba948" (UID: "d857cd33-2469-441a-9ac0-d1c6b27ba948"). InnerVolumeSpecName "kube-api-access-b4bfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:25:36.981438 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.981382 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-dshm" pod "d857cd33-2469-441a-9ac0-d1c6b27ba948" (UID: "d857cd33-2469-441a-9ac0-d1c6b27ba948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:36.981569 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:36.981546 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d857cd33-2469-441a-9ac0-d1c6b27ba948-tls-certs" pod "d857cd33-2469-441a-9ac0-d1c6b27ba948" (UID: "d857cd33-2469-441a-9ac0-d1c6b27ba948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:25:37.079158 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.079116 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4bfz\" (UniqueName: \"kubernetes.io/projected/d857cd33-2469-441a-9ac0-d1c6b27ba948-kube-api-access-b4bfz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:37.079449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.079427 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:37.079594 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.079578 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:37.079696 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.079684 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:37.079798 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.079787 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d857cd33-2469-441a-9ac0-d1c6b27ba948-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:37.079896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.079885 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d857cd33-2469-441a-9ac0-d1c6b27ba948-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:37.781529 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.781498 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" Nov 01 21:25:37.781529 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.781513 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r" event={"ID":"d857cd33-2469-441a-9ac0-d1c6b27ba948","Type":"ContainerDied","Data":"7e007fc94703e18242c68cfb475f13aa45d7b5846862f59604f4ef7d9c15085c"} Nov 01 21:25:37.782006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.781560 5633 scope.go:122] "RemoveContainer" containerID="c784092370b367e9afe17728851fe7c6216603f9bd0b4fcbe97bfb1dbed639e1" Nov 01 21:25:37.961215 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:37.961180 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r"] Nov 01 21:25:38.011429 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:38.011387 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9cd91242-kserve-f6cd4cc85-8dz4r"] Nov 01 21:25:38.466932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:38.466896 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d857cd33-2469-441a-9ac0-d1c6b27ba948" path="/var/lib/kubelet/pods/d857cd33-2469-441a-9ac0-d1c6b27ba948/volumes" Nov 01 21:25:41.185667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:41.185632 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:25:45.784167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:45.784135 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:25:46.157591 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:25:46.157499 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:25:51.567114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:51.567031 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:25:58.019554 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.019521 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f"] Nov 01 21:25:58.020068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.020054 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d857cd33-2469-441a-9ac0-d1c6b27ba948" containerName="main" Nov 01 21:25:58.020119 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.020071 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d857cd33-2469-441a-9ac0-d1c6b27ba948" containerName="main" Nov 01 21:25:58.020169 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.020160 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d857cd33-2469-441a-9ac0-d1c6b27ba948" containerName="main" Nov 01 21:25:58.023450 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.023431 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.078437 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.078399 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f"] Nov 01 21:25:58.102847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.102798 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.103062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.102943 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wkw\" (UniqueName: \"kubernetes.io/projected/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-kube-api-access-q8wkw\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.103062 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.103029 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.124180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.124147 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:25:58.124180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.124155 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:25:58.124440 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.124204 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:25:58.125532 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.125509 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:25:58.204300 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.204259 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.204518 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.204325 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wkw\" (UniqueName: \"kubernetes.io/projected/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-kube-api-access-q8wkw\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.204518 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.204402 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.205026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.205005 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.206888 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.206871 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.260424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.260379 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wkw\" (UniqueName: \"kubernetes.io/projected/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-kube-api-access-q8wkw\") pod \"maas-api-revoke-sub-3df379a0-l9g4f\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.337559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.337473 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:25:58.710422 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:58.710394 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f"] Nov 01 21:25:58.711971 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:25:58.711946 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556d5ea4_e415_4e1f_8d6e_68ebb24587b9.slice/crio-ad1dfac80291d2f9de87b798cc60378bd10362fdd1c3b71f0d7f96a38d5fbf7c WatchSource:0}: Error finding container ad1dfac80291d2f9de87b798cc60378bd10362fdd1c3b71f0d7f96a38d5fbf7c: Status 404 returned error can't find the container with id ad1dfac80291d2f9de87b798cc60378bd10362fdd1c3b71f0d7f96a38d5fbf7c Nov 01 21:25:59.002898 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:25:59.002870 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" event={"ID":"556d5ea4-e415-4e1f-8d6e-68ebb24587b9","Type":"ContainerStarted","Data":"ad1dfac80291d2f9de87b798cc60378bd10362fdd1c3b71f0d7f96a38d5fbf7c"} Nov 01 21:26:00.010906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:00.010869 5633 generic.go:356] "Generic (PLEG): container finished" podID="556d5ea4-e415-4e1f-8d6e-68ebb24587b9" containerID="1a3157a288172f586aacd3d2c1ef56fb1cd0f24f7075e407e348dda6022ee1bf" exitCode=0 Nov 01 21:26:00.011413 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:00.010907 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" event={"ID":"556d5ea4-e415-4e1f-8d6e-68ebb24587b9","Type":"ContainerDied","Data":"1a3157a288172f586aacd3d2c1ef56fb1cd0f24f7075e407e348dda6022ee1bf"} Nov 01 21:26:01.158735 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.158705 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:26:01.239766 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.239732 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-kube-api-access-q8wkw\" (UniqueName: \"kubernetes.io/projected/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-kube-api-access-q8wkw\") pod \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " Nov 01 21:26:01.239946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.239774 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-openshift-ca\") pod \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " Nov 01 21:26:01.239946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.239827 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-xks-ca\") pod \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\" (UID: \"556d5ea4-e415-4e1f-8d6e-68ebb24587b9\") " Nov 01 21:26:01.240206 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.240174 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-openshift-ca" pod "556d5ea4-e415-4e1f-8d6e-68ebb24587b9" (UID: "556d5ea4-e415-4e1f-8d6e-68ebb24587b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:26:01.242207 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.242182 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-xks-ca" pod "556d5ea4-e415-4e1f-8d6e-68ebb24587b9" (UID: "556d5ea4-e415-4e1f-8d6e-68ebb24587b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:26:01.242207 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.242190 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-kube-api-access-q8wkw" pod "556d5ea4-e415-4e1f-8d6e-68ebb24587b9" (UID: "556d5ea4-e415-4e1f-8d6e-68ebb24587b9"). InnerVolumeSpecName "kube-api-access-q8wkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:26:01.341708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.341618 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wkw\" (UniqueName: \"kubernetes.io/projected/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-kube-api-access-q8wkw\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:01.341708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.341653 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:01.341708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:01.341664 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556d5ea4-e415-4e1f-8d6e-68ebb24587b9-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:26:02.023296 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:02.023268 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" Nov 01 21:26:02.023514 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:02.023309 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f" event={"ID":"556d5ea4-e415-4e1f-8d6e-68ebb24587b9","Type":"ContainerDied","Data":"ad1dfac80291d2f9de87b798cc60378bd10362fdd1c3b71f0d7f96a38d5fbf7c"} Nov 01 21:26:02.023514 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:02.023399 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad1dfac80291d2f9de87b798cc60378bd10362fdd1c3b71f0d7f96a38d5fbf7c" Nov 01 21:26:10.963096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:10.963056 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:26:13.648764 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:13.648732 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:14.337380 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:14.337344 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:14.462480 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:14.462448 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:16.157380 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:26:16.157314 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:26:18.688293 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:18.688260 5633 scope.go:122] "RemoveContainer" containerID="091d6b92cf2f000c110aea833160ab3741f3b1a3dc699f36898d67243f14e569" Nov 01 21:26:18.691077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:18.691041 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:18.876901 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:18.876863 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:19.002852 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:19.002815 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:21.875132 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:21.875096 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:22.045593 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:22.045541 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:23.269646 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:23.269610 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:23.439303 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:23.439263 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:30.603739 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:30.603707 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:30.769442 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:30.769401 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:30.883722 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:30.883622 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:34.224058 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:34.224023 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:34.700821 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:34.700736 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:35.812828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.812794 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj"] Nov 01 21:26:35.813297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.813283 5633 cpu_manager.go:414] "containerMap: removing container" podUID="556d5ea4-e415-4e1f-8d6e-68ebb24587b9" containerName="revoke-keys" Nov 01 21:26:35.813375 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.813299 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556d5ea4-e415-4e1f-8d6e-68ebb24587b9" containerName="revoke-keys" Nov 01 21:26:35.813423 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.813414 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="556d5ea4-e415-4e1f-8d6e-68ebb24587b9" containerName="revoke-keys" Nov 01 21:26:35.816757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.816742 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:35.872775 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.872744 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj"] Nov 01 21:26:35.918143 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.918108 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-958b82\"" Nov 01 21:26:35.918370 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.918187 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-958b82-dockercfg-rsrkq\"" Nov 01 21:26:35.920789 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.920753 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcpt\" (UniqueName: \"kubernetes.io/projected/1493e474-6955-413b-82a6-b6422b239e5b-kube-api-access-fhcpt\") pod \"payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:35.920970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:35.920942 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1493e474-6955-413b-82a6-b6422b239e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:36.022320 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.022281 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1493e474-6955-413b-82a6-b6422b239e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:36.022544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.022404 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcpt\" (UniqueName: \"kubernetes.io/projected/1493e474-6955-413b-82a6-b6422b239e5b-kube-api-access-fhcpt\") pod \"payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:36.023093 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.023073 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1493e474-6955-413b-82a6-b6422b239e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:36.079028 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.078954 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcpt\" (UniqueName: \"kubernetes.io/projected/1493e474-6955-413b-82a6-b6422b239e5b-kube-api-access-fhcpt\") pod \"payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:36.130690 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.130662 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:36.512195 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.511868 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj"] Nov 01 21:26:36.520783 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.517473 5633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:26:36.623482 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.623444 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp"] Nov 01 21:26:36.628643 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.628610 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:36.660167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.660123 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:26:36.683921 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.683877 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp"] Nov 01 21:26:36.729731 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.729688 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdf243-728a-470b-9433-be6c4aca22ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:36.729950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.729890 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7kpf\" (UniqueName: \"kubernetes.io/projected/f8bdf243-728a-470b-9433-be6c4aca22ae-kube-api-access-j7kpf\") pod \"payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:36.831644 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.831561 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7kpf\" (UniqueName: \"kubernetes.io/projected/f8bdf243-728a-470b-9433-be6c4aca22ae-kube-api-access-j7kpf\") pod \"payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:36.832021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.831664 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdf243-728a-470b-9433-be6c4aca22ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:36.832647 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.832622 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdf243-728a-470b-9433-be6c4aca22ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:36.887732 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.887696 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7kpf\" (UniqueName: \"kubernetes.io/projected/f8bdf243-728a-470b-9433-be6c4aca22ae-kube-api-access-j7kpf\") pod \"payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:36.949430 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:36.949397 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:37.119648 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:37.119548 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:37.237788 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:37.237749 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" event={"ID":"1493e474-6955-413b-82a6-b6422b239e5b","Type":"ContainerStarted","Data":"771a0722e4b1e1ca7fe0a52bc3591eb6bc7f60113ec9d938cb49071409fa582c"} Nov 01 21:26:37.326189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:37.326154 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp"] Nov 01 21:26:37.327243 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:26:37.327218 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bdf243_728a_470b_9433_be6c4aca22ae.slice/crio-3371f0dcbc3a871cad03c490573f08ccb12a4089d0425b3ad06ca8d49e0f3f67 WatchSource:0}: Error finding container 3371f0dcbc3a871cad03c490573f08ccb12a4089d0425b3ad06ca8d49e0f3f67: Status 404 returned error can't find the container with id 3371f0dcbc3a871cad03c490573f08ccb12a4089d0425b3ad06ca8d49e0f3f67 Nov 01 21:26:38.246530 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:38.246483 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" event={"ID":"f8bdf243-728a-470b-9433-be6c4aca22ae","Type":"ContainerStarted","Data":"3371f0dcbc3a871cad03c490573f08ccb12a4089d0425b3ad06ca8d49e0f3f67"} Nov 01 21:26:41.266478 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:41.266444 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" event={"ID":"f8bdf243-728a-470b-9433-be6c4aca22ae","Type":"ContainerStarted","Data":"1c38f3f410a0d48b1d7a29f8e18c3d1da493a25fbbc8f336d72eeea114802e7e"} Nov 01 21:26:41.266936 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:41.266533 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:41.268020 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:41.267993 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" event={"ID":"1493e474-6955-413b-82a6-b6422b239e5b","Type":"ContainerStarted","Data":"de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530"} Nov 01 21:26:41.268166 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:41.268096 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:41.380110 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:41.380051 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" podStartSLOduration=2.436729288 podStartE2EDuration="5.38003726s" totalImagesPullingTime="2.943307972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:41.378057388 +0000 UTC m=+5009.052316643" watchObservedRunningTime="2026-11-01 21:26:41.38003726 +0000 UTC m=+5009.054296514" Nov 01 21:26:41.490206 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:41.490123 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" podStartSLOduration=2.737294201 podStartE2EDuration="6.49010384s" totalImagesPullingTime="3.752809639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:41.488619344 +0000 UTC m=+5009.162878600" watchObservedRunningTime="2026-11-01 21:26:41.49010384 +0000 UTC m=+5009.164363095" Nov 01 21:26:41.989446 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:41.989407 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:46.115542 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:46.115503 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:46.156776 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:26:46.156741 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:26:48.274313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:48.274274 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:48.926160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:48.926115 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:52.275484 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:52.275455 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:26:52.275968 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:52.275528 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:26:53.480775 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:53.480733 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:54.912767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:54.912728 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:55.554549 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:55.554510 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:55.727997 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:55.727891 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:59.872313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:59.872272 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:26:59.992102 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:26:59.992063 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:00.118486 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:00.118445 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:03.048150 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:03.048110 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:03.209365 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:03.209292 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:04.579744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:04.579714 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:04.748477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:04.748448 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:05.111834 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:05.111804 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:11.584886 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:11.584843 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:11.767769 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:11.767732 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:11.889043 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:11.888946 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:15.200075 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:15.200046 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:15.672681 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:15.672646 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:16.158852 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:27:16.158812 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:27:16.789506 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.789469 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t"] Nov 01 21:27:16.793801 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.793780 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:16.850285 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.850249 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t"] Nov 01 21:27:16.851186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.851151 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1192b51c-90a8-4169-b397-85fa88013988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:16.851475 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.851455 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjlsc\" (UniqueName: \"kubernetes.io/projected/1192b51c-90a8-4169-b397-85fa88013988-kube-api-access-fjlsc\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:16.894996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.894961 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30b7a8\"" Nov 01 21:27:16.895219 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.895091 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30b7a8-dockercfg-jrfgd\"" Nov 01 21:27:16.952794 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.952750 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1192b51c-90a8-4169-b397-85fa88013988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:16.952984 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.952807 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlsc\" (UniqueName: \"kubernetes.io/projected/1192b51c-90a8-4169-b397-85fa88013988-kube-api-access-fjlsc\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:16.953515 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:16.953495 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1192b51c-90a8-4169-b397-85fa88013988-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:17.009371 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.009313 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlsc\" (UniqueName: \"kubernetes.io/projected/1192b51c-90a8-4169-b397-85fa88013988-kube-api-access-fjlsc\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:17.107399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.107281 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:17.486092 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.486071 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t"] Nov 01 21:27:17.488186 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:27:17.488155 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1192b51c_90a8_4169_b397_85fa88013988.slice/crio-c157578f2c766839fa3b1e134a1c571a8106bfa5c082990ae339ca9410e71510 WatchSource:0}: Error finding container c157578f2c766839fa3b1e134a1c571a8106bfa5c082990ae339ca9410e71510: Status 404 returned error can't find the container with id c157578f2c766839fa3b1e134a1c571a8106bfa5c082990ae339ca9410e71510 Nov 01 21:27:17.594017 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.593981 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm"] Nov 01 21:27:17.598632 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.598607 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:17.651114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.651070 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm"] Nov 01 21:27:17.663972 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.663860 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226dfb4a-2b5c-4007-925f-e434f0fa5f73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:17.664189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.664013 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258pd\" (UniqueName: \"kubernetes.io/projected/226dfb4a-2b5c-4007-925f-e434f0fa5f73-kube-api-access-258pd\") pod \"payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:17.765567 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.765467 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226dfb4a-2b5c-4007-925f-e434f0fa5f73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:17.765761 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.765614 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258pd\" (UniqueName: \"kubernetes.io/projected/226dfb4a-2b5c-4007-925f-e434f0fa5f73-kube-api-access-258pd\") pod \"payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:17.766287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.766254 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226dfb4a-2b5c-4007-925f-e434f0fa5f73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:17.821721 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.821690 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-258pd\" (UniqueName: \"kubernetes.io/projected/226dfb4a-2b5c-4007-925f-e434f0fa5f73-kube-api-access-258pd\") pod \"payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:17.892445 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.892397 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:27:17.914172 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:17.914138 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:18.186529 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.186432 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:18.294163 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.294129 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm"] Nov 01 21:27:18.295450 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:27:18.295420 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226dfb4a_2b5c_4007_925f_e434f0fa5f73.slice/crio-124c238a02fb91791063760b166d3e5ebad699fb429a4e5dbc8cc92d18d84e94 WatchSource:0}: Error finding container 124c238a02fb91791063760b166d3e5ebad699fb429a4e5dbc8cc92d18d84e94: Status 404 returned error can't find the container with id 124c238a02fb91791063760b166d3e5ebad699fb429a4e5dbc8cc92d18d84e94 Nov 01 21:27:18.492121 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.492087 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" event={"ID":"1192b51c-90a8-4169-b397-85fa88013988","Type":"ContainerStarted","Data":"8dd1a6ad2e76e0058e71a86c4406f0de51111c0e9ac9c86ab177730c776e7d38"} Nov 01 21:27:18.492385 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.492144 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" event={"ID":"1192b51c-90a8-4169-b397-85fa88013988","Type":"ContainerStarted","Data":"c157578f2c766839fa3b1e134a1c571a8106bfa5c082990ae339ca9410e71510"} Nov 01 21:27:18.492385 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.492262 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:18.493930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.493904 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" event={"ID":"226dfb4a-2b5c-4007-925f-e434f0fa5f73","Type":"ContainerStarted","Data":"1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03"} Nov 01 21:27:18.494045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.493945 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" event={"ID":"226dfb4a-2b5c-4007-925f-e434f0fa5f73","Type":"ContainerStarted","Data":"124c238a02fb91791063760b166d3e5ebad699fb429a4e5dbc8cc92d18d84e94"} Nov 01 21:27:18.494045 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.493965 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:18.607291 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.607231 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" podStartSLOduration=2.607215095 podStartE2EDuration="2.607215095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:27:18.603904358 +0000 UTC m=+5046.278163611" watchObservedRunningTime="2026-11-01 21:27:18.607215095 +0000 UTC m=+5046.281474350" Nov 01 21:27:18.719416 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:18.718702 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" podStartSLOduration=1.7186748729999999 podStartE2EDuration="1.718674873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:27:18.71646939 +0000 UTC m=+5046.390728642" watchObservedRunningTime="2026-11-01 21:27:18.718674873 +0000 UTC m=+5046.392934161" Nov 01 21:27:23.127810 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:23.127718 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:27.599733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:27.599696 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:29.502554 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:29.502519 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:27:29.502965 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:29.502703 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:27:29.521644 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:29.521611 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:33.980753 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:33.980710 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:34.102262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:34.102215 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:35.770323 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:35.770277 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:40.044588 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:40.044553 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld"] Nov 01 21:27:40.093478 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:40.093440 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3fe42-5j6ld"] Nov 01 21:27:40.397519 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:40.397422 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:40.466415 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:40.466381 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd1f6b2-2380-4c5a-8f39-30198da8f197" path="/var/lib/kubelet/pods/4cd1f6b2-2380-4c5a-8f39-30198da8f197/volumes" Nov 01 21:27:40.886596 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:40.886555 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:46.156527 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:27:46.156489 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:27:46.568564 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:46.568527 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:47.467610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:47.467579 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm_226dfb4a-2b5c-4007-925f-e434f0fa5f73/payload-processing/0.log" Nov 01 21:27:47.828394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:47.828287 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:52.201393 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:52.201360 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:53.684896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:53.684862 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:27:56.801477 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:56.801438 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:27:58.586133 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:27:58.586089 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:05.000232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:05.000183 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:11.296674 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:11.296639 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:16.102235 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:16.102192 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:16.156816 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:28:16.156768 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:28:20.305305 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:20.305258 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:20.426114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:20.426078 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:22.577584 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:22.577542 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:28.879953 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:28.879915 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:33.771974 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:33.771932 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:40.183075 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:40.183034 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:40.853853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:40.853812 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:28:46.157407 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:28:46.157364 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:28:46.567581 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:46.567529 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:46.799257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:46.799215 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:48.133720 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:48.133687 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp_f8bdf243-728a-470b-9433-be6c4aca22ae/payload-processing/0.log" Nov 01 21:28:48.812237 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:48.812208 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm_226dfb4a-2b5c-4007-925f-e434f0fa5f73/payload-processing/0.log" Nov 01 21:28:49.033622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:49.033591 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm_226dfb4a-2b5c-4007-925f-e434f0fa5f73/payload-processing/0.log" Nov 01 21:28:49.999642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:49.999600 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:51.372090 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:51.372005 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:55.058377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:55.058318 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:28:57.779022 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:28:57.778978 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:04.082037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:04.082003 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:08.964881 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:08.964831 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:15.382938 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:15.382894 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:16.156380 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:29:16.156312 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:29:18.739260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:18.739225 5633 scope.go:122] "RemoveContainer" containerID="2f33584c275a781f4d5bfe1579cbe8434a9f11e088b337d54fb252631fbf7dd0" Nov 01 21:29:21.675567 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:21.675529 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:26.570915 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:26.570881 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:31.585440 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:31.585403 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:32.036417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.036373 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s"] Nov 01 21:29:32.040189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.040163 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.097262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.097222 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s"] Nov 01 21:29:32.110254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.110205 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.110452 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.110354 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj86\" (UniqueName: \"kubernetes.io/projected/be974d7c-410d-4d34-a33e-3228354bbe8c-kube-api-access-fwj86\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.110502 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.110488 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.141879 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.141838 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:29:32.141879 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.141838 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:29:32.142120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.141921 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:29:32.142120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.141950 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:29:32.190047 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.189996 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:32.211261 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.211222 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj86\" (UniqueName: \"kubernetes.io/projected/be974d7c-410d-4d34-a33e-3228354bbe8c-kube-api-access-fwj86\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.211553 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.211378 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.211553 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.211482 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.212209 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.212173 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.214190 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.214169 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.267769 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.267731 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj86\" (UniqueName: \"kubernetes.io/projected/be974d7c-410d-4d34-a33e-3228354bbe8c-kube-api-access-fwj86\") pod \"maas-api-revoke-sub-00d62574-6pf5s\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.353813 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.353716 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:32.732368 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:32.732316 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s"] Nov 01 21:29:32.733852 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:29:32.733822 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe974d7c_410d_4d34_a33e_3228354bbe8c.slice/crio-02df137f2bcd3cb4b063b4315562fa5dd2f9926b994c2d31269e45cbab3937fe WatchSource:0}: Error finding container 02df137f2bcd3cb4b063b4315562fa5dd2f9926b994c2d31269e45cbab3937fe: Status 404 returned error can't find the container with id 02df137f2bcd3cb4b063b4315562fa5dd2f9926b994c2d31269e45cbab3937fe Nov 01 21:29:33.225553 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:33.225518 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:33.292813 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:33.292690 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" event={"ID":"be974d7c-410d-4d34-a33e-3228354bbe8c","Type":"ContainerStarted","Data":"2c780fa346b1e7bd836a7dfba6d2ba7f98d74118e002a048aa05cd8b2494a560"} Nov 01 21:29:33.292813 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:33.292756 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" event={"ID":"be974d7c-410d-4d34-a33e-3228354bbe8c","Type":"ContainerStarted","Data":"02df137f2bcd3cb4b063b4315562fa5dd2f9926b994c2d31269e45cbab3937fe"} Nov 01 21:29:33.407173 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:33.407084 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" podStartSLOduration=2.407062516 podStartE2EDuration="2.407062516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:29:33.404912376 +0000 UTC m=+5181.079171632" watchObservedRunningTime="2026-11-01 21:29:33.407062516 +0000 UTC m=+5181.081321772" Nov 01 21:29:34.300033 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:34.299995 5633 generic.go:356] "Generic (PLEG): container finished" podID="be974d7c-410d-4d34-a33e-3228354bbe8c" containerID="2c780fa346b1e7bd836a7dfba6d2ba7f98d74118e002a048aa05cd8b2494a560" exitCode=0 Nov 01 21:29:34.300478 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:34.300081 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" event={"ID":"be974d7c-410d-4d34-a33e-3228354bbe8c","Type":"ContainerDied","Data":"2c780fa346b1e7bd836a7dfba6d2ba7f98d74118e002a048aa05cd8b2494a560"} Nov 01 21:29:35.459850 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.459824 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:35.548613 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.548568 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be974d7c-410d-4d34-a33e-3228354bbe8c-kube-api-access-fwj86\" (UniqueName: \"kubernetes.io/projected/be974d7c-410d-4d34-a33e-3228354bbe8c-kube-api-access-fwj86\") pod \"be974d7c-410d-4d34-a33e-3228354bbe8c\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " Nov 01 21:29:35.548613 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.548620 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-xks-ca\") pod \"be974d7c-410d-4d34-a33e-3228354bbe8c\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " Nov 01 21:29:35.548886 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.548677 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-openshift-ca\") pod \"be974d7c-410d-4d34-a33e-3228354bbe8c\" (UID: \"be974d7c-410d-4d34-a33e-3228354bbe8c\") " Nov 01 21:29:35.549063 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.549039 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-openshift-ca" pod "be974d7c-410d-4d34-a33e-3228354bbe8c" (UID: "be974d7c-410d-4d34-a33e-3228354bbe8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:29:35.550946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.550912 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-xks-ca" pod "be974d7c-410d-4d34-a33e-3228354bbe8c" (UID: "be974d7c-410d-4d34-a33e-3228354bbe8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:29:35.550946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.550928 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be974d7c-410d-4d34-a33e-3228354bbe8c-kube-api-access-fwj86" pod "be974d7c-410d-4d34-a33e-3228354bbe8c" (UID: "be974d7c-410d-4d34-a33e-3228354bbe8c"). InnerVolumeSpecName "kube-api-access-fwj86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:29:35.650444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.650403 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwj86\" (UniqueName: \"kubernetes.io/projected/be974d7c-410d-4d34-a33e-3228354bbe8c-kube-api-access-fwj86\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:35.650444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.650440 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:35.650444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:35.650450 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be974d7c-410d-4d34-a33e-3228354bbe8c-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:36.173638 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:36.173594 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:29:36.315177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:36.315141 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" event={"ID":"be974d7c-410d-4d34-a33e-3228354bbe8c","Type":"ContainerDied","Data":"02df137f2bcd3cb4b063b4315562fa5dd2f9926b994c2d31269e45cbab3937fe"} Nov 01 21:29:36.315177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:36.315178 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s" Nov 01 21:29:36.315436 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:36.315193 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02df137f2bcd3cb4b063b4315562fa5dd2f9926b994c2d31269e45cbab3937fe" Nov 01 21:29:37.579597 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:37.579563 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:39.575552 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:39.575482 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:43.126082 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:43.126048 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:43.661404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:43.661372 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:44.014667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.014632 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp"] Nov 01 21:29:44.014899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.014877 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" podUID="f8bdf243-728a-470b-9433-be6c4aca22ae" containerName="payload-processing" containerID="cri-o://1c38f3f410a0d48b1d7a29f8e18c3d1da493a25fbbc8f336d72eeea114802e7e" gracePeriod=30 Nov 01 21:29:44.066729 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.066698 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj"] Nov 01 21:29:44.066907 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.066885 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" podUID="1493e474-6955-413b-82a6-b6422b239e5b" containerName="payload-pre-processing" containerID="cri-o://de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530" gracePeriod=30 Nov 01 21:29:44.157823 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.157788 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:44.328153 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.328113 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:44.336840 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.336813 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:29:44.361180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.361151 5633 generic.go:356] "Generic (PLEG): container finished" podID="f8bdf243-728a-470b-9433-be6c4aca22ae" containerID="1c38f3f410a0d48b1d7a29f8e18c3d1da493a25fbbc8f336d72eeea114802e7e" exitCode=0 Nov 01 21:29:44.361321 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.361224 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" event={"ID":"f8bdf243-728a-470b-9433-be6c4aca22ae","Type":"ContainerDied","Data":"1c38f3f410a0d48b1d7a29f8e18c3d1da493a25fbbc8f336d72eeea114802e7e"} Nov 01 21:29:44.362667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.362646 5633 generic.go:356] "Generic (PLEG): container finished" podID="1493e474-6955-413b-82a6-b6422b239e5b" containerID="de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530" exitCode=0 Nov 01 21:29:44.362790 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.362698 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" Nov 01 21:29:44.362790 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.362729 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" event={"ID":"1493e474-6955-413b-82a6-b6422b239e5b","Type":"ContainerDied","Data":"de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530"} Nov 01 21:29:44.362790 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.362784 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj" event={"ID":"1493e474-6955-413b-82a6-b6422b239e5b","Type":"ContainerDied","Data":"771a0722e4b1e1ca7fe0a52bc3591eb6bc7f60113ec9d938cb49071409fa582c"} Nov 01 21:29:44.362973 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.362815 5633 scope.go:122] "RemoveContainer" containerID="de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530" Nov 01 21:29:44.376798 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.376780 5633 scope.go:122] "RemoveContainer" containerID="de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530" Nov 01 21:29:44.377064 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:29:44.377045 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530\": container with ID starting with de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530 not found: ID does not exist" containerID="de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530" Nov 01 21:29:44.377146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.377071 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530"} err="failed to get container status \"de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530\": rpc error: code = NotFound desc = could not find container \"de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530\": container with ID starting with de520a3206582b6529dbfb23cbd63af0eb6058759e42e50e89ae896ee9079530 not found: ID does not exist" Nov 01 21:29:44.378217 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.378197 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:29:44.447882 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.447849 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bdf243-728a-470b-9433-be6c4aca22ae-kube-api-access-j7kpf\" (UniqueName: \"kubernetes.io/projected/f8bdf243-728a-470b-9433-be6c4aca22ae-kube-api-access-j7kpf\") pod \"f8bdf243-728a-470b-9433-be6c4aca22ae\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " Nov 01 21:29:44.448059 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.447958 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1493e474-6955-413b-82a6-b6422b239e5b-kube-api-access-fhcpt\" (UniqueName: \"kubernetes.io/projected/1493e474-6955-413b-82a6-b6422b239e5b-kube-api-access-fhcpt\") pod \"1493e474-6955-413b-82a6-b6422b239e5b\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " Nov 01 21:29:44.448059 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.448010 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1493e474-6955-413b-82a6-b6422b239e5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1493e474-6955-413b-82a6-b6422b239e5b-plugins-config-volume\") pod \"1493e474-6955-413b-82a6-b6422b239e5b\" (UID: \"1493e474-6955-413b-82a6-b6422b239e5b\") " Nov 01 21:29:44.448059 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.448046 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bdf243-728a-470b-9433-be6c4aca22ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdf243-728a-470b-9433-be6c4aca22ae-plugins-config-volume\") pod \"f8bdf243-728a-470b-9433-be6c4aca22ae\" (UID: \"f8bdf243-728a-470b-9433-be6c4aca22ae\") " Nov 01 21:29:44.448484 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.448457 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1493e474-6955-413b-82a6-b6422b239e5b-plugins-config-volume" pod "1493e474-6955-413b-82a6-b6422b239e5b" (UID: "1493e474-6955-413b-82a6-b6422b239e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:29:44.448602 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.448506 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bdf243-728a-470b-9433-be6c4aca22ae-plugins-config-volume" pod "f8bdf243-728a-470b-9433-be6c4aca22ae" (UID: "f8bdf243-728a-470b-9433-be6c4aca22ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:29:44.451546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.451514 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bdf243-728a-470b-9433-be6c4aca22ae-kube-api-access-j7kpf" pod "f8bdf243-728a-470b-9433-be6c4aca22ae" (UID: "f8bdf243-728a-470b-9433-be6c4aca22ae"). InnerVolumeSpecName "kube-api-access-j7kpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:29:44.451673 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.451555 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1493e474-6955-413b-82a6-b6422b239e5b-kube-api-access-fhcpt" pod "1493e474-6955-413b-82a6-b6422b239e5b" (UID: "1493e474-6955-413b-82a6-b6422b239e5b"). InnerVolumeSpecName "kube-api-access-fhcpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:29:44.548838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.548804 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7kpf\" (UniqueName: \"kubernetes.io/projected/f8bdf243-728a-470b-9433-be6c4aca22ae-kube-api-access-j7kpf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:44.548838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.548834 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcpt\" (UniqueName: \"kubernetes.io/projected/1493e474-6955-413b-82a6-b6422b239e5b-kube-api-access-fhcpt\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:44.548838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.548843 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1493e474-6955-413b-82a6-b6422b239e5b-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:44.549094 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.548852 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bdf243-728a-470b-9433-be6c4aca22ae-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:44.776696 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.776661 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj"] Nov 01 21:29:44.780219 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:44.780188 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-7d9b94bfff-79dfj"] Nov 01 21:29:45.370860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:45.370761 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" Nov 01 21:29:45.370860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:45.370766 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp" event={"ID":"f8bdf243-728a-470b-9433-be6c4aca22ae","Type":"ContainerDied","Data":"3371f0dcbc3a871cad03c490573f08ccb12a4089d0425b3ad06ca8d49e0f3f67"} Nov 01 21:29:45.371415 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:45.370898 5633 scope.go:122] "RemoveContainer" containerID="1c38f3f410a0d48b1d7a29f8e18c3d1da493a25fbbc8f336d72eeea114802e7e" Nov 01 21:29:45.534168 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:45.534130 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp"] Nov 01 21:29:45.539679 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:45.539649 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-757c9c5966-h52pp"] Nov 01 21:29:45.584753 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:45.584712 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:46.157030 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:29:46.156989 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:29:46.466146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:46.466118 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1493e474-6955-413b-82a6-b6422b239e5b" path="/var/lib/kubelet/pods/1493e474-6955-413b-82a6-b6422b239e5b/volumes" Nov 01 21:29:46.466589 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:46.466538 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bdf243-728a-470b-9433-be6c4aca22ae" path="/var/lib/kubelet/pods/f8bdf243-728a-470b-9433-be6c4aca22ae/volumes" Nov 01 21:29:47.143645 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:47.143598 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:48.647966 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:48.647928 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:48.817204 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:48.817166 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:48.961970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:48.961941 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:29:50.163109 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:50.163075 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:50.755794 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:50.755760 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:52.128826 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:52.128789 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:56.122233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:56.122140 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:57.181457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:57.181408 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:58.008449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:58.008397 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:29:58.176460 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:29:58.176424 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:00.163087 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163053 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g"] Nov 01 21:30:00.163650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163568 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f8bdf243-728a-470b-9433-be6c4aca22ae" containerName="payload-processing" Nov 01 21:30:00.163650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163582 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bdf243-728a-470b-9433-be6c4aca22ae" containerName="payload-processing" Nov 01 21:30:00.163650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163599 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1493e474-6955-413b-82a6-b6422b239e5b" containerName="payload-pre-processing" Nov 01 21:30:00.163650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163604 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1493e474-6955-413b-82a6-b6422b239e5b" containerName="payload-pre-processing" Nov 01 21:30:00.163650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163613 5633 cpu_manager.go:414] "containerMap: removing container" podUID="be974d7c-410d-4d34-a33e-3228354bbe8c" containerName="revoke-keys" Nov 01 21:30:00.163650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163618 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be974d7c-410d-4d34-a33e-3228354bbe8c" containerName="revoke-keys" Nov 01 21:30:00.163995 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163698 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bdf243-728a-470b-9433-be6c4aca22ae" containerName="payload-processing" Nov 01 21:30:00.163995 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163710 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1493e474-6955-413b-82a6-b6422b239e5b" containerName="payload-pre-processing" Nov 01 21:30:00.163995 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.163717 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="be974d7c-410d-4d34-a33e-3228354bbe8c" containerName="revoke-keys" Nov 01 21:30:00.167030 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.167010 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl"] Nov 01 21:30:00.167181 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.167164 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.171283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.171264 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.222429 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.222396 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g"] Nov 01 21:30:00.267024 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.266985 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:30:00.267024 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.267023 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:30:00.267249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.267023 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:30:00.273714 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.273681 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl"] Nov 01 21:30:00.314197 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.314159 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892810-wrd6g\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.314448 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.314201 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.314448 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.314235 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vkp\" (UniqueName: \"kubernetes.io/projected/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-kube-api-access-c7vkp\") pod \"maas-api-key-cleanup-29892810-wrd6g\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.314600 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.314435 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfj4r\" (UniqueName: \"kubernetes.io/projected/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-kube-api-access-gfj4r\") pod \"maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.415999 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.415900 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892810-wrd6g\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.415999 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.415944 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.416252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.416071 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vkp\" (UniqueName: \"kubernetes.io/projected/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-kube-api-access-c7vkp\") pod \"maas-api-key-cleanup-29892810-wrd6g\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.416252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.416148 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfj4r\" (UniqueName: \"kubernetes.io/projected/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-kube-api-access-gfj4r\") pod \"maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.416743 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.416719 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892810-wrd6g\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.416838 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.416745 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.472423 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.472384 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vkp\" (UniqueName: \"kubernetes.io/projected/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-kube-api-access-c7vkp\") pod \"maas-api-key-cleanup-29892810-wrd6g\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.472590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.472383 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfj4r\" (UniqueName: \"kubernetes.io/projected/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-kube-api-access-gfj4r\") pod \"maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.482111 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.482083 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:00.488451 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.488419 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:00.860819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.860787 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g"] Nov 01 21:30:00.861152 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:30:00.861129 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfa4dc3_9294_436b_9358_5c1b1db4ff1b.slice/crio-e031a300221738579463e62d1e600402492860e5161ec07845332ca3dd293804 WatchSource:0}: Error finding container e031a300221738579463e62d1e600402492860e5161ec07845332ca3dd293804: Status 404 returned error can't find the container with id e031a300221738579463e62d1e600402492860e5161ec07845332ca3dd293804 Nov 01 21:30:00.917533 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:00.911893 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl"] Nov 01 21:30:01.466732 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:01.466700 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" event={"ID":"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b","Type":"ContainerStarted","Data":"3a882e7c2bae3668b2a1ad497ca9debd26052706d6da97e4559262516bd1ac3e"} Nov 01 21:30:01.467237 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:01.466751 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" event={"ID":"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b","Type":"ContainerStarted","Data":"e031a300221738579463e62d1e600402492860e5161ec07845332ca3dd293804"} Nov 01 21:30:01.468377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:01.468345 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" event={"ID":"a8b5989b-d1a3-44d4-b0ce-6036f78384c0","Type":"ContainerStarted","Data":"98ad60cd92f6d7bad86355676cf621e75f51a196320ae0a197d18548fb2f10e1"} Nov 01 21:30:01.468528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:01.468401 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" event={"ID":"a8b5989b-d1a3-44d4-b0ce-6036f78384c0","Type":"ContainerStarted","Data":"cc4d66c20f3f32f995b95599887075c42f98e29bc86e798b0e4ba754dd4ca0c8"} Nov 01 21:30:01.579502 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:01.579437 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" podStartSLOduration=1.579420466 podStartE2EDuration="1.579420466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:01.577226367 +0000 UTC m=+5209.251485619" watchObservedRunningTime="2026-11-01 21:30:01.579420466 +0000 UTC m=+5209.253679724" Nov 01 21:30:01.689437 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:01.689382 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" podStartSLOduration=1.6893667369999998 podStartE2EDuration="1.689366737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:01.688263067 +0000 UTC m=+5209.362522319" watchObservedRunningTime="2026-11-01 21:30:01.689366737 +0000 UTC m=+5209.363625991" Nov 01 21:30:01.873509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:01.873454 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:02.475642 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:02.475602 5633 generic.go:356] "Generic (PLEG): container finished" podID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerID="3a882e7c2bae3668b2a1ad497ca9debd26052706d6da97e4559262516bd1ac3e" exitCode=60 Nov 01 21:30:02.476134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:02.475689 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" event={"ID":"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b","Type":"ContainerDied","Data":"3a882e7c2bae3668b2a1ad497ca9debd26052706d6da97e4559262516bd1ac3e"} Nov 01 21:30:02.476134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:02.475955 5633 scope.go:122] "RemoveContainer" containerID="3a882e7c2bae3668b2a1ad497ca9debd26052706d6da97e4559262516bd1ac3e" Nov 01 21:30:02.479092 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:02.479063 5633 generic.go:356] "Generic (PLEG): container finished" podID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerID="98ad60cd92f6d7bad86355676cf621e75f51a196320ae0a197d18548fb2f10e1" exitCode=60 Nov 01 21:30:02.479219 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:02.479172 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" event={"ID":"a8b5989b-d1a3-44d4-b0ce-6036f78384c0","Type":"ContainerDied","Data":"98ad60cd92f6d7bad86355676cf621e75f51a196320ae0a197d18548fb2f10e1"} Nov 01 21:30:02.479508 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:02.479485 5633 scope.go:122] "RemoveContainer" containerID="98ad60cd92f6d7bad86355676cf621e75f51a196320ae0a197d18548fb2f10e1" Nov 01 21:30:03.486963 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.486927 5633 generic.go:356] "Generic (PLEG): container finished" podID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerID="e45aefb3c82b6e0c194a22fb81b4cfdc4bab0845835687d1c8dc3179cc0f6b93" exitCode=60 Nov 01 21:30:03.487536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.487071 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" event={"ID":"a8b5989b-d1a3-44d4-b0ce-6036f78384c0","Type":"ContainerDied","Data":"e45aefb3c82b6e0c194a22fb81b4cfdc4bab0845835687d1c8dc3179cc0f6b93"} Nov 01 21:30:03.487536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.487131 5633 scope.go:122] "RemoveContainer" containerID="98ad60cd92f6d7bad86355676cf621e75f51a196320ae0a197d18548fb2f10e1" Nov 01 21:30:03.487536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.487388 5633 scope.go:122] "RemoveContainer" containerID="e45aefb3c82b6e0c194a22fb81b4cfdc4bab0845835687d1c8dc3179cc0f6b93" Nov 01 21:30:03.487691 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:03.487667 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl_odh-ai-gateway-infra(a8b5989b-d1a3-44d4-b0ce-6036f78384c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" Nov 01 21:30:03.490033 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.490010 5633 generic.go:356] "Generic (PLEG): container finished" podID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerID="72c6dab7e2fab77b019d398019689c756ac3aa744757101dbf30d9c1c5b52548" exitCode=60 Nov 01 21:30:03.490152 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.490061 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" event={"ID":"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b","Type":"ContainerDied","Data":"72c6dab7e2fab77b019d398019689c756ac3aa744757101dbf30d9c1c5b52548"} Nov 01 21:30:03.490356 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.490311 5633 scope.go:122] "RemoveContainer" containerID="72c6dab7e2fab77b019d398019689c756ac3aa744757101dbf30d9c1c5b52548" Nov 01 21:30:03.490587 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:03.490571 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892810-wrd6g_odh-ai-gateway-infra(4dfa4dc3-9294-436b-9358-5c1b1db4ff1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" Nov 01 21:30:03.500372 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.500320 5633 scope.go:122] "RemoveContainer" containerID="3a882e7c2bae3668b2a1ad497ca9debd26052706d6da97e4559262516bd1ac3e" Nov 01 21:30:03.543212 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:03.543171 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:04.498625 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:04.498596 5633 scope.go:122] "RemoveContainer" containerID="72c6dab7e2fab77b019d398019689c756ac3aa744757101dbf30d9c1c5b52548" Nov 01 21:30:04.499095 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:04.498928 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892810-wrd6g_odh-ai-gateway-infra(4dfa4dc3-9294-436b-9358-5c1b1db4ff1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" Nov 01 21:30:04.500296 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:04.500265 5633 scope.go:122] "RemoveContainer" containerID="e45aefb3c82b6e0c194a22fb81b4cfdc4bab0845835687d1c8dc3179cc0f6b93" Nov 01 21:30:04.500552 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:04.500523 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl_odh-ai-gateway-infra(a8b5989b-d1a3-44d4-b0ce-6036f78384c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" Nov 01 21:30:07.830399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:07.830349 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:07.920525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:07.920486 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl"] Nov 01 21:30:08.076840 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.076811 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:08.184673 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.184586 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm"] Nov 01 21:30:08.184835 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.184796 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" podUID="226dfb4a-2b5c-4007-925f-e434f0fa5f73" containerName="payload-processing" containerID="cri-o://1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03" gracePeriod=30 Nov 01 21:30:08.203878 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.203852 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-maas-api-service-ca\") pod \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " Nov 01 21:30:08.204568 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.203918 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-kube-api-access-gfj4r\" (UniqueName: \"kubernetes.io/projected/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-kube-api-access-gfj4r\") pod \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\" (UID: \"a8b5989b-d1a3-44d4-b0ce-6036f78384c0\") " Nov 01 21:30:08.204568 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.204254 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-maas-api-service-ca" pod "a8b5989b-d1a3-44d4-b0ce-6036f78384c0" (UID: "a8b5989b-d1a3-44d4-b0ce-6036f78384c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:30:08.204568 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.204417 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:08.207055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.207032 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-kube-api-access-gfj4r" pod "a8b5989b-d1a3-44d4-b0ce-6036f78384c0" (UID: "a8b5989b-d1a3-44d4-b0ce-6036f78384c0"). InnerVolumeSpecName "kube-api-access-gfj4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:30:08.239399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.239362 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t"] Nov 01 21:30:08.239631 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.239608 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" podUID="1192b51c-90a8-4169-b397-85fa88013988" containerName="payload-pre-processing" containerID="cri-o://8dd1a6ad2e76e0058e71a86c4406f0de51111c0e9ac9c86ab177730c776e7d38" gracePeriod=30 Nov 01 21:30:08.305689 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.305657 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfj4r\" (UniqueName: \"kubernetes.io/projected/a8b5989b-d1a3-44d4-b0ce-6036f78384c0-kube-api-access-gfj4r\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:08.325860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.325817 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:08.444669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.444580 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:08.492721 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.492696 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:30:08.526065 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.526038 5633 generic.go:356] "Generic (PLEG): container finished" podID="226dfb4a-2b5c-4007-925f-e434f0fa5f73" containerID="1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03" exitCode=0 Nov 01 21:30:08.526161 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.526091 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" Nov 01 21:30:08.526161 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.526121 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" event={"ID":"226dfb4a-2b5c-4007-925f-e434f0fa5f73","Type":"ContainerDied","Data":"1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03"} Nov 01 21:30:08.526417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.526181 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm" event={"ID":"226dfb4a-2b5c-4007-925f-e434f0fa5f73","Type":"ContainerDied","Data":"124c238a02fb91791063760b166d3e5ebad699fb429a4e5dbc8cc92d18d84e94"} Nov 01 21:30:08.526417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.526211 5633 scope.go:122] "RemoveContainer" containerID="1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03" Nov 01 21:30:08.527961 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.527941 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" Nov 01 21:30:08.527961 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.527952 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl" event={"ID":"a8b5989b-d1a3-44d4-b0ce-6036f78384c0","Type":"ContainerDied","Data":"cc4d66c20f3f32f995b95599887075c42f98e29bc86e798b0e4ba754dd4ca0c8"} Nov 01 21:30:08.529647 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.529620 5633 generic.go:356] "Generic (PLEG): container finished" podID="1192b51c-90a8-4169-b397-85fa88013988" containerID="8dd1a6ad2e76e0058e71a86c4406f0de51111c0e9ac9c86ab177730c776e7d38" exitCode=0 Nov 01 21:30:08.529783 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.529692 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" event={"ID":"1192b51c-90a8-4169-b397-85fa88013988","Type":"ContainerDied","Data":"8dd1a6ad2e76e0058e71a86c4406f0de51111c0e9ac9c86ab177730c776e7d38"} Nov 01 21:30:08.536658 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.536634 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:30:08.537150 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.537134 5633 scope.go:122] "RemoveContainer" containerID="1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03" Nov 01 21:30:08.537428 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:08.537408 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03\": container with ID starting with 1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03 not found: ID does not exist" containerID="1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03" Nov 01 21:30:08.537486 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.537438 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03"} err="failed to get container status \"1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03\": rpc error: code = NotFound desc = could not find container \"1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03\": container with ID starting with 1873ff466ae52536df4e85236a5f7d9497452717300953e3f454a92a5afd1c03 not found: ID does not exist" Nov 01 21:30:08.537486 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.537455 5633 scope.go:122] "RemoveContainer" containerID="e45aefb3c82b6e0c194a22fb81b4cfdc4bab0845835687d1c8dc3179cc0f6b93" Nov 01 21:30:08.614891 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.614860 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226dfb4a-2b5c-4007-925f-e434f0fa5f73-kube-api-access-258pd\" (UniqueName: \"kubernetes.io/projected/226dfb4a-2b5c-4007-925f-e434f0fa5f73-kube-api-access-258pd\") pod \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " Nov 01 21:30:08.615447 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.615413 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226dfb4a-2b5c-4007-925f-e434f0fa5f73-plugins-config-volume" pod "226dfb4a-2b5c-4007-925f-e434f0fa5f73" (UID: "226dfb4a-2b5c-4007-925f-e434f0fa5f73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:30:08.615594 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.615106 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226dfb4a-2b5c-4007-925f-e434f0fa5f73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226dfb4a-2b5c-4007-925f-e434f0fa5f73-plugins-config-volume\") pod \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\" (UID: \"226dfb4a-2b5c-4007-925f-e434f0fa5f73\") " Nov 01 21:30:08.615854 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.615838 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226dfb4a-2b5c-4007-925f-e434f0fa5f73-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:08.617521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.617501 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226dfb4a-2b5c-4007-925f-e434f0fa5f73-kube-api-access-258pd" pod "226dfb4a-2b5c-4007-925f-e434f0fa5f73" (UID: "226dfb4a-2b5c-4007-925f-e434f0fa5f73"). InnerVolumeSpecName "kube-api-access-258pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:30:08.714611 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.714574 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl"] Nov 01 21:30:08.716360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.716304 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1192b51c-90a8-4169-b397-85fa88013988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1192b51c-90a8-4169-b397-85fa88013988-plugins-config-volume\") pod \"1192b51c-90a8-4169-b397-85fa88013988\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " Nov 01 21:30:08.716519 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.716393 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1192b51c-90a8-4169-b397-85fa88013988-kube-api-access-fjlsc\" (UniqueName: \"kubernetes.io/projected/1192b51c-90a8-4169-b397-85fa88013988-kube-api-access-fjlsc\") pod \"1192b51c-90a8-4169-b397-85fa88013988\" (UID: \"1192b51c-90a8-4169-b397-85fa88013988\") " Nov 01 21:30:08.716683 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.716660 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1192b51c-90a8-4169-b397-85fa88013988-plugins-config-volume" pod "1192b51c-90a8-4169-b397-85fa88013988" (UID: "1192b51c-90a8-4169-b397-85fa88013988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:30:08.716830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.716813 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1192b51c-90a8-4169-b397-85fa88013988-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:08.716889 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.716838 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258pd\" (UniqueName: \"kubernetes.io/projected/226dfb4a-2b5c-4007-925f-e434f0fa5f73-kube-api-access-258pd\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:08.718676 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.718646 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1192b51c-90a8-4169-b397-85fa88013988-kube-api-access-fjlsc" pod "1192b51c-90a8-4169-b397-85fa88013988" (UID: "1192b51c-90a8-4169-b397-85fa88013988"). InnerVolumeSpecName "kube-api-access-fjlsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:30:08.718745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.718707 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-30b7a8-29892810-jqbvl"] Nov 01 21:30:08.817794 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.817757 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjlsc\" (UniqueName: \"kubernetes.io/projected/1192b51c-90a8-4169-b397-85fa88013988-kube-api-access-fjlsc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:08.988907 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.988815 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm"] Nov 01 21:30:08.993364 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:08.993317 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-9d9bb4974-j5pkm"] Nov 01 21:30:09.538889 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:09.538850 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" event={"ID":"1192b51c-90a8-4169-b397-85fa88013988","Type":"ContainerDied","Data":"c157578f2c766839fa3b1e134a1c571a8106bfa5c082990ae339ca9410e71510"} Nov 01 21:30:09.539090 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:09.538909 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t" Nov 01 21:30:09.539090 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:09.538926 5633 scope.go:122] "RemoveContainer" containerID="8dd1a6ad2e76e0058e71a86c4406f0de51111c0e9ac9c86ab177730c776e7d38" Nov 01 21:30:09.705471 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:09.705425 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t"] Nov 01 21:30:09.709135 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:09.709106 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-6bb8f867f-cf58t"] Nov 01 21:30:10.468648 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:10.468607 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1192b51c-90a8-4169-b397-85fa88013988" path="/var/lib/kubelet/pods/1192b51c-90a8-4169-b397-85fa88013988/volumes" Nov 01 21:30:10.469194 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:10.469173 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226dfb4a-2b5c-4007-925f-e434f0fa5f73" path="/var/lib/kubelet/pods/226dfb4a-2b5c-4007-925f-e434f0fa5f73/volumes" Nov 01 21:30:10.471557 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:10.471527 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" path="/var/lib/kubelet/pods/a8b5989b-d1a3-44d4-b0ce-6036f78384c0/volumes" Nov 01 21:30:10.567230 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:10.567192 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:30:14.899674 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:14.899635 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:15.018350 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:15.018310 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:15.697891 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:15.697848 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:15.830466 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:15.830424 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:16.157967 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:16.157929 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:30:17.241065 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:17.241017 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:17.708607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:17.708469 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:30:18.460193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:18.460158 5633 scope.go:122] "RemoveContainer" containerID="72c6dab7e2fab77b019d398019689c756ac3aa744757101dbf30d9c1c5b52548" Nov 01 21:30:18.765949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:18.765920 5633 scope.go:122] "RemoveContainer" containerID="ba53f1cd6fa411ac94b1da063cf84a7633fedb1c74c12e8edae6ba303f6c727e" Nov 01 21:30:19.599363 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:19.599254 5633 generic.go:356] "Generic (PLEG): container finished" podID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerID="d904eab16d5b9b65500ca6ac52cca09fcc2f3befe2f367e18bf36492bbec24e4" exitCode=60 Nov 01 21:30:19.599363 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:19.599323 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" event={"ID":"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b","Type":"ContainerDied","Data":"d904eab16d5b9b65500ca6ac52cca09fcc2f3befe2f367e18bf36492bbec24e4"} Nov 01 21:30:19.599819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:19.599424 5633 scope.go:122] "RemoveContainer" containerID="72c6dab7e2fab77b019d398019689c756ac3aa744757101dbf30d9c1c5b52548" Nov 01 21:30:19.599819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:19.599760 5633 scope.go:122] "RemoveContainer" containerID="d904eab16d5b9b65500ca6ac52cca09fcc2f3befe2f367e18bf36492bbec24e4" Nov 01 21:30:19.600059 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:19.600040 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892810-wrd6g_odh-ai-gateway-infra(4dfa4dc3-9294-436b-9358-5c1b1db4ff1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" Nov 01 21:30:20.723806 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:20.723766 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g"] Nov 01 21:30:20.876669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:20.876638 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:20.937349 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:20.937304 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-maas-api-service-ca\") pod \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " Nov 01 21:30:20.937504 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:20.937417 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-kube-api-access-c7vkp\" (UniqueName: \"kubernetes.io/projected/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-kube-api-access-c7vkp\") pod \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\" (UID: \"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b\") " Nov 01 21:30:20.937761 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:20.937738 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-maas-api-service-ca" pod "4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" (UID: "4dfa4dc3-9294-436b-9358-5c1b1db4ff1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:30:20.939715 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:20.939681 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-kube-api-access-c7vkp" pod "4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" (UID: "4dfa4dc3-9294-436b-9358-5c1b1db4ff1b"). InnerVolumeSpecName "kube-api-access-c7vkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:30:21.038790 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:21.038681 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vkp\" (UniqueName: \"kubernetes.io/projected/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-kube-api-access-c7vkp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:21.038790 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:21.038709 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:21.615024 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:21.614988 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" event={"ID":"4dfa4dc3-9294-436b-9358-5c1b1db4ff1b","Type":"ContainerDied","Data":"e031a300221738579463e62d1e600402492860e5161ec07845332ca3dd293804"} Nov 01 21:30:21.615024 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:21.615024 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g" Nov 01 21:30:21.615323 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:21.615063 5633 scope.go:122] "RemoveContainer" containerID="d904eab16d5b9b65500ca6ac52cca09fcc2f3befe2f367e18bf36492bbec24e4" Nov 01 21:30:21.777173 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:21.777135 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g"] Nov 01 21:30:21.780315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:21.780283 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892810-wrd6g"] Nov 01 21:30:22.467245 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:22.467214 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" path="/var/lib/kubelet/pods/4dfa4dc3-9294-436b-9358-5c1b1db4ff1b/volumes" Nov 01 21:30:31.418992 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:31.418952 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s75g6"] Nov 01 21:30:36.420260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.420186 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" podUID="cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" containerName="payload-pre-processing" containerID="cri-o://e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07" gracePeriod=25 Nov 01 21:30:36.670476 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.670404 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 21:30:36.698770 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.698737 5633 generic.go:356] "Generic (PLEG): container finished" podID="cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" containerID="e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07" exitCode=0 Nov 01 21:30:36.698951 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.698838 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" Nov 01 21:30:36.698951 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.698883 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" event={"ID":"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb","Type":"ContainerDied","Data":"e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07"} Nov 01 21:30:36.698951 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.698937 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-s75g6" event={"ID":"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb","Type":"ContainerDied","Data":"37d3acbee1be4919d440fef8576247c30dab21485176fde3d831badeea8e595c"} Nov 01 21:30:36.699064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.698960 5633 scope.go:122] "RemoveContainer" containerID="e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07" Nov 01 21:30:36.708294 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.708275 5633 scope.go:122] "RemoveContainer" containerID="e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07" Nov 01 21:30:36.708647 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:36.708627 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07\": container with ID starting with e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07 not found: ID does not exist" containerID="e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07" Nov 01 21:30:36.708726 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.708656 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07"} err="failed to get container status \"e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07\": rpc error: code = NotFound desc = could not find container \"e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07\": container with ID starting with e1c760ee5ecfc43da23bc39aa1e5146656fd8e8c8ba0f3acf46f37a267ba1e07 not found: ID does not exist" Nov 01 21:30:36.803108 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.803076 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-plugins-config-volume\") pod \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " Nov 01 21:30:36.803294 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.803116 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-kube-api-access-4dhwb\" (UniqueName: \"kubernetes.io/projected/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-kube-api-access-4dhwb\") pod \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\" (UID: \"cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb\") " Nov 01 21:30:36.803582 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.803559 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-plugins-config-volume" pod "cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" (UID: "cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:30:36.805496 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.805467 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-kube-api-access-4dhwb" pod "cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" (UID: "cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb"). InnerVolumeSpecName "kube-api-access-4dhwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:30:36.904875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.904835 5633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-plugins-config-volume\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:36.904875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:36.904866 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhwb\" (UniqueName: \"kubernetes.io/projected/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb-kube-api-access-4dhwb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:30:37.118179 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:37.118144 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s75g6"] Nov 01 21:30:37.121255 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:37.121229 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-s75g6"] Nov 01 21:30:38.466310 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:38.466276 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" path="/var/lib/kubelet/pods/cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb/volumes" Nov 01 21:30:46.156505 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:30:46.156459 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:30:47.511526 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:30:47.511488 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:02.047912 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:02.047878 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f"] Nov 01 21:31:02.096649 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:02.096610 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df379a0-l9g4f"] Nov 01 21:31:02.466752 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:02.466716 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556d5ea4-e415-4e1f-8d6e-68ebb24587b9" path="/var/lib/kubelet/pods/556d5ea4-e415-4e1f-8d6e-68ebb24587b9/volumes" Nov 01 21:31:03.234835 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:03.234796 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:03.671534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:03.671442 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:06.277162 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:06.277130 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:31:08.083793 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:08.083754 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:31:16.155524 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:31:16.155479 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:31:46.156956 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:31:46.156899 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:31:47.938950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:31:47.938916 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:32:16.156613 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:32:16.156572 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:32:18.833491 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:18.833455 5633 scope.go:122] "RemoveContainer" containerID="1a3157a288172f586aacd3d2c1ef56fb1cd0f24f7075e407e348dda6022ee1bf" Nov 01 21:32:46.156894 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:32:46.156859 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:32:48.210360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:48.210299 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:32:53.739786 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.739740 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22"] Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740210 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740222 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740231 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740239 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740261 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740267 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740277 5633 cpu_manager.go:414] "containerMap: removing container" podUID="1192b51c-90a8-4169-b397-85fa88013988" containerName="payload-pre-processing" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740281 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1192b51c-90a8-4169-b397-85fa88013988" containerName="payload-pre-processing" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740290 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740295 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerName="cleanup" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740310 5633 cpu_manager.go:414] "containerMap: removing container" podUID="cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" containerName="payload-pre-processing" Nov 01 21:32:53.740307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740314 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" containerName="payload-pre-processing" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740325 5633 cpu_manager.go:414] "containerMap: removing container" podUID="226dfb4a-2b5c-4007-925f-e434f0fa5f73" containerName="payload-processing" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740347 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226dfb4a-2b5c-4007-925f-e434f0fa5f73" containerName="payload-processing" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740421 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1192b51c-90a8-4169-b397-85fa88013988" containerName="payload-pre-processing" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740428 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerName="cleanup" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740437 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb80d241-7c98-4cc5-a861-eb5e8cb2c9fb" containerName="payload-pre-processing" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740443 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="226dfb4a-2b5c-4007-925f-e434f0fa5f73" containerName="payload-processing" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740450 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740456 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:32:53.740741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.740464 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:32:53.743800 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.743779 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.795610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.795577 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:32:53.844865 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.844831 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:32:53.845055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.844873 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:32:53.845055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.845030 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:32:53.846313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.846282 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.846471 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.846414 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.846516 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.846488 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5pv\" (UniqueName: \"kubernetes.io/projected/c3fa631e-256b-4c6e-a83a-36c857b40528-kube-api-access-2v5pv\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.848168 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.848148 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22"] Nov 01 21:32:53.947314 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.947281 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.947525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.947361 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5pv\" (UniqueName: \"kubernetes.io/projected/c3fa631e-256b-4c6e-a83a-36c857b40528-kube-api-access-2v5pv\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.947525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.947449 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.948171 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.948142 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:53.949980 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:53.949951 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:54.003391 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:54.003306 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5pv\" (UniqueName: \"kubernetes.io/projected/c3fa631e-256b-4c6e-a83a-36c857b40528-kube-api-access-2v5pv\") pod \"maas-api-revoke-sub-e2547622-x5l22\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:54.056930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:54.056897 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:54.430979 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:54.430947 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22"] Nov 01 21:32:54.432974 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:32:54.432935 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fa631e_256b_4c6e_a83a_36c857b40528.slice/crio-1c63533534831707f38f2b0da8de1d44c6008f9d0c58b36f3dc1cbedba5a7fb1 WatchSource:0}: Error finding container 1c63533534831707f38f2b0da8de1d44c6008f9d0c58b36f3dc1cbedba5a7fb1: Status 404 returned error can't find the container with id 1c63533534831707f38f2b0da8de1d44c6008f9d0c58b36f3dc1cbedba5a7fb1 Nov 01 21:32:54.566543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:54.566497 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" event={"ID":"c3fa631e-256b-4c6e-a83a-36c857b40528","Type":"ContainerStarted","Data":"1c63533534831707f38f2b0da8de1d44c6008f9d0c58b36f3dc1cbedba5a7fb1"} Nov 01 21:32:55.573938 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:55.573842 5633 generic.go:356] "Generic (PLEG): container finished" podID="c3fa631e-256b-4c6e-a83a-36c857b40528" containerID="04369fde8da0b7a3497bd2695b87f716bd5bdbc172e3309932273cbf67ef8d87" exitCode=0 Nov 01 21:32:55.574406 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:55.573929 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" event={"ID":"c3fa631e-256b-4c6e-a83a-36c857b40528","Type":"ContainerDied","Data":"04369fde8da0b7a3497bd2695b87f716bd5bdbc172e3309932273cbf67ef8d87"} Nov 01 21:32:56.724633 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.724607 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:56.879070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.878972 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-xks-ca\") pod \"c3fa631e-256b-4c6e-a83a-36c857b40528\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " Nov 01 21:32:56.879070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.879061 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-openshift-ca\") pod \"c3fa631e-256b-4c6e-a83a-36c857b40528\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " Nov 01 21:32:56.879320 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.879093 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fa631e-256b-4c6e-a83a-36c857b40528-kube-api-access-2v5pv\" (UniqueName: \"kubernetes.io/projected/c3fa631e-256b-4c6e-a83a-36c857b40528-kube-api-access-2v5pv\") pod \"c3fa631e-256b-4c6e-a83a-36c857b40528\" (UID: \"c3fa631e-256b-4c6e-a83a-36c857b40528\") " Nov 01 21:32:56.879462 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.879438 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-openshift-ca" pod "c3fa631e-256b-4c6e-a83a-36c857b40528" (UID: "c3fa631e-256b-4c6e-a83a-36c857b40528"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:32:56.881235 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.881207 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-xks-ca" pod "c3fa631e-256b-4c6e-a83a-36c857b40528" (UID: "c3fa631e-256b-4c6e-a83a-36c857b40528"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:32:56.881394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.881370 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fa631e-256b-4c6e-a83a-36c857b40528-kube-api-access-2v5pv" pod "c3fa631e-256b-4c6e-a83a-36c857b40528" (UID: "c3fa631e-256b-4c6e-a83a-36c857b40528"). InnerVolumeSpecName "kube-api-access-2v5pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:32:56.980292 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.980251 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v5pv\" (UniqueName: \"kubernetes.io/projected/c3fa631e-256b-4c6e-a83a-36c857b40528-kube-api-access-2v5pv\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:32:56.980292 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.980286 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:32:56.980292 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:56.980296 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fa631e-256b-4c6e-a83a-36c857b40528-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:32:57.586579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:57.586551 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" Nov 01 21:32:57.586769 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:57.586581 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22" event={"ID":"c3fa631e-256b-4c6e-a83a-36c857b40528","Type":"ContainerDied","Data":"1c63533534831707f38f2b0da8de1d44c6008f9d0c58b36f3dc1cbedba5a7fb1"} Nov 01 21:32:57.586769 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:32:57.586626 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c63533534831707f38f2b0da8de1d44c6008f9d0c58b36f3dc1cbedba5a7fb1" Nov 01 21:33:05.329300 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:05.329259 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:33:05.478445 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:05.478409 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:33:05.849060 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:05.849023 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:33:08.356450 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:08.356418 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:33:16.156696 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:33:16.156660 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:33:18.990314 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:18.990278 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:33:25.552052 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.551967 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd"] Nov 01 21:33:25.552590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.552455 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:33:25.552590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.552467 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfa4dc3-9294-436b-9358-5c1b1db4ff1b" containerName="cleanup" Nov 01 21:33:25.552590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.552496 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c3fa631e-256b-4c6e-a83a-36c857b40528" containerName="revoke-keys" Nov 01 21:33:25.552590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.552500 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fa631e-256b-4c6e-a83a-36c857b40528" containerName="revoke-keys" Nov 01 21:33:25.552758 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.552593 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fa631e-256b-4c6e-a83a-36c857b40528" containerName="revoke-keys" Nov 01 21:33:25.552758 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.552605 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5989b-d1a3-44d4-b0ce-6036f78384c0" containerName="cleanup" Nov 01 21:33:25.557076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.557050 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.611157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.611122 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd"] Nov 01 21:33:25.656546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.656511 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:33:25.656546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.656511 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:33:25.656546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.656546 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:33:25.658177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.658158 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:33:25.669853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.669818 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsxd\" (UniqueName: \"kubernetes.io/projected/cb62c07d-6972-4deb-918c-542185c30db8-kube-api-access-kbsxd\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.670287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.669971 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.670287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.670044 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.771694 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.771641 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.771898 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.771711 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsxd\" (UniqueName: \"kubernetes.io/projected/cb62c07d-6972-4deb-918c-542185c30db8-kube-api-access-kbsxd\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.771898 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.771790 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.772366 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.772309 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.774360 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.774316 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.828152 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.828074 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsxd\" (UniqueName: \"kubernetes.io/projected/cb62c07d-6972-4deb-918c-542185c30db8-kube-api-access-kbsxd\") pod \"maas-api-revoke-sub-c1e271a9-746sd\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:25.871949 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:25.871910 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:26.244669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:26.244645 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd"] Nov 01 21:33:26.246136 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:33:26.246105 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb62c07d_6972_4deb_918c_542185c30db8.slice/crio-942e5ff09ad6a1168a236b9bffdea765d1e9de80c32203a6749b85085b82971c WatchSource:0}: Error finding container 942e5ff09ad6a1168a236b9bffdea765d1e9de80c32203a6749b85085b82971c: Status 404 returned error can't find the container with id 942e5ff09ad6a1168a236b9bffdea765d1e9de80c32203a6749b85085b82971c Nov 01 21:33:26.792523 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:26.792434 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" event={"ID":"cb62c07d-6972-4deb-918c-542185c30db8","Type":"ContainerStarted","Data":"23c76b0672e8c9cacb4a6539557bd5d96cdf351e5fff6a56e59c6bab30a77fb5"} Nov 01 21:33:26.792523 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:26.792491 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" event={"ID":"cb62c07d-6972-4deb-918c-542185c30db8","Type":"ContainerStarted","Data":"942e5ff09ad6a1168a236b9bffdea765d1e9de80c32203a6749b85085b82971c"} Nov 01 21:33:26.905778 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:26.905720 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" podStartSLOduration=1.905707496 podStartE2EDuration="1.905707496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:33:26.90523353 +0000 UTC m=+5414.579492786" watchObservedRunningTime="2026-11-01 21:33:26.905707496 +0000 UTC m=+5414.579966751" Nov 01 21:33:27.800161 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:27.800124 5633 generic.go:356] "Generic (PLEG): container finished" podID="cb62c07d-6972-4deb-918c-542185c30db8" containerID="23c76b0672e8c9cacb4a6539557bd5d96cdf351e5fff6a56e59c6bab30a77fb5" exitCode=0 Nov 01 21:33:27.800636 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:27.800183 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" event={"ID":"cb62c07d-6972-4deb-918c-542185c30db8","Type":"ContainerDied","Data":"23c76b0672e8c9cacb4a6539557bd5d96cdf351e5fff6a56e59c6bab30a77fb5"} Nov 01 21:33:28.954643 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:28.954608 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:29.002953 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.002914 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb62c07d-6972-4deb-918c-542185c30db8-kube-api-access-kbsxd\" (UniqueName: \"kubernetes.io/projected/cb62c07d-6972-4deb-918c-542185c30db8-kube-api-access-kbsxd\") pod \"cb62c07d-6972-4deb-918c-542185c30db8\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " Nov 01 21:33:29.003172 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.002984 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-xks-ca\") pod \"cb62c07d-6972-4deb-918c-542185c30db8\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " Nov 01 21:33:29.003172 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.003045 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-openshift-ca\") pod \"cb62c07d-6972-4deb-918c-542185c30db8\" (UID: \"cb62c07d-6972-4deb-918c-542185c30db8\") " Nov 01 21:33:29.003501 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.003462 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-openshift-ca" pod "cb62c07d-6972-4deb-918c-542185c30db8" (UID: "cb62c07d-6972-4deb-918c-542185c30db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:33:29.005386 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.005359 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-xks-ca" pod "cb62c07d-6972-4deb-918c-542185c30db8" (UID: "cb62c07d-6972-4deb-918c-542185c30db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:33:29.005503 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.005427 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb62c07d-6972-4deb-918c-542185c30db8-kube-api-access-kbsxd" pod "cb62c07d-6972-4deb-918c-542185c30db8" (UID: "cb62c07d-6972-4deb-918c-542185c30db8"). InnerVolumeSpecName "kube-api-access-kbsxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:33:29.104064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.103977 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsxd\" (UniqueName: \"kubernetes.io/projected/cb62c07d-6972-4deb-918c-542185c30db8-kube-api-access-kbsxd\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:29.104064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.104010 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:29.104064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.104021 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb62c07d-6972-4deb-918c-542185c30db8-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:29.812724 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.812692 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" Nov 01 21:33:29.812906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.812730 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd" event={"ID":"cb62c07d-6972-4deb-918c-542185c30db8","Type":"ContainerDied","Data":"942e5ff09ad6a1168a236b9bffdea765d1e9de80c32203a6749b85085b82971c"} Nov 01 21:33:29.812906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:29.812779 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="942e5ff09ad6a1168a236b9bffdea765d1e9de80c32203a6749b85085b82971c" Nov 01 21:33:38.386869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:38.386833 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:33:46.218360 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:33:46.215563 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:33:47.443172 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:47.443117 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:33:47.443737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:47.443203 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:33:48.293443 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:48.293407 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:33:48.586600 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:48.586506 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:33:50.039055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.039014 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv"] Nov 01 21:33:50.039688 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.039671 5633 cpu_manager.go:414] "containerMap: removing container" podUID="cb62c07d-6972-4deb-918c-542185c30db8" containerName="revoke-keys" Nov 01 21:33:50.039735 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.039691 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb62c07d-6972-4deb-918c-542185c30db8" containerName="revoke-keys" Nov 01 21:33:50.039831 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.039820 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb62c07d-6972-4deb-918c-542185c30db8" containerName="revoke-keys" Nov 01 21:33:50.043521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.043498 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.097449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.097407 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv"] Nov 01 21:33:50.137625 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.137593 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.137801 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.137657 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.137801 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.137745 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvp6\" (UniqueName: \"kubernetes.io/projected/60288308-2e6f-4621-bac3-0483cbd3aa1f-kube-api-access-qcvp6\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.143094 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.143060 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:33:50.143094 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.143083 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:33:50.143296 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.143089 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:33:50.144526 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.144511 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:33:50.239409 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.239365 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.239616 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.239454 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.239616 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.239482 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvp6\" (UniqueName: \"kubernetes.io/projected/60288308-2e6f-4621-bac3-0483cbd3aa1f-kube-api-access-qcvp6\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.240176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.240150 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.241943 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.241924 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.296006 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.295924 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvp6\" (UniqueName: \"kubernetes.io/projected/60288308-2e6f-4621-bac3-0483cbd3aa1f-kube-api-access-qcvp6\") pod \"maas-api-revoke-sub-2e1f5b5f-wq7qv\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.358366 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.358301 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:50.734306 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.734277 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv"] Nov 01 21:33:50.736013 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:33:50.735981 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60288308_2e6f_4621_bac3_0483cbd3aa1f.slice/crio-8fa34fca5823b1a311a5e2b1e6a382cd4675f9813be60d3963ed305f66830205 WatchSource:0}: Error finding container 8fa34fca5823b1a311a5e2b1e6a382cd4675f9813be60d3963ed305f66830205: Status 404 returned error can't find the container with id 8fa34fca5823b1a311a5e2b1e6a382cd4675f9813be60d3963ed305f66830205 Nov 01 21:33:50.982059 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:50.982012 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" event={"ID":"60288308-2e6f-4621-bac3-0483cbd3aa1f","Type":"ContainerStarted","Data":"8fa34fca5823b1a311a5e2b1e6a382cd4675f9813be60d3963ed305f66830205"} Nov 01 21:33:51.988994 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:51.988954 5633 generic.go:356] "Generic (PLEG): container finished" podID="60288308-2e6f-4621-bac3-0483cbd3aa1f" containerID="c9bbde6660187efde2f2a80a7696a9d4eb7aa670d594b10cefd79f697ee1a628" exitCode=0 Nov 01 21:33:51.989417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:51.989028 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" event={"ID":"60288308-2e6f-4621-bac3-0483cbd3aa1f","Type":"ContainerDied","Data":"c9bbde6660187efde2f2a80a7696a9d4eb7aa670d594b10cefd79f697ee1a628"} Nov 01 21:33:53.139037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.139011 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:53.271168 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.271079 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-openshift-ca\") pod \"60288308-2e6f-4621-bac3-0483cbd3aa1f\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " Nov 01 21:33:53.271168 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.271127 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-xks-ca\") pod \"60288308-2e6f-4621-bac3-0483cbd3aa1f\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " Nov 01 21:33:53.271168 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.271165 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60288308-2e6f-4621-bac3-0483cbd3aa1f-kube-api-access-qcvp6\" (UniqueName: \"kubernetes.io/projected/60288308-2e6f-4621-bac3-0483cbd3aa1f-kube-api-access-qcvp6\") pod \"60288308-2e6f-4621-bac3-0483cbd3aa1f\" (UID: \"60288308-2e6f-4621-bac3-0483cbd3aa1f\") " Nov 01 21:33:53.271581 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.271554 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-openshift-ca" pod "60288308-2e6f-4621-bac3-0483cbd3aa1f" (UID: "60288308-2e6f-4621-bac3-0483cbd3aa1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:33:53.271705 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.271686 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:53.273517 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.273487 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-xks-ca" pod "60288308-2e6f-4621-bac3-0483cbd3aa1f" (UID: "60288308-2e6f-4621-bac3-0483cbd3aa1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:33:53.273697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.273675 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60288308-2e6f-4621-bac3-0483cbd3aa1f-kube-api-access-qcvp6" pod "60288308-2e6f-4621-bac3-0483cbd3aa1f" (UID: "60288308-2e6f-4621-bac3-0483cbd3aa1f"). InnerVolumeSpecName "kube-api-access-qcvp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:33:53.373187 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.373141 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60288308-2e6f-4621-bac3-0483cbd3aa1f-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:53.373187 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:53.373178 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvp6\" (UniqueName: \"kubernetes.io/projected/60288308-2e6f-4621-bac3-0483cbd3aa1f-kube-api-access-qcvp6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:33:54.001024 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:54.000993 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" Nov 01 21:33:54.001024 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:54.001009 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv" event={"ID":"60288308-2e6f-4621-bac3-0483cbd3aa1f","Type":"ContainerDied","Data":"8fa34fca5823b1a311a5e2b1e6a382cd4675f9813be60d3963ed305f66830205"} Nov 01 21:33:54.001275 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:33:54.001072 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa34fca5823b1a311a5e2b1e6a382cd4675f9813be60d3963ed305f66830205" Nov 01 21:34:02.467699 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.467665 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4"] Nov 01 21:34:02.468130 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.468090 5633 cpu_manager.go:414] "containerMap: removing container" podUID="60288308-2e6f-4621-bac3-0483cbd3aa1f" containerName="revoke-keys" Nov 01 21:34:02.468130 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.468101 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60288308-2e6f-4621-bac3-0483cbd3aa1f" containerName="revoke-keys" Nov 01 21:34:02.468212 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.468167 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="60288308-2e6f-4621-bac3-0483cbd3aa1f" containerName="revoke-keys" Nov 01 21:34:02.471366 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.471342 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.525129 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.525087 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4"] Nov 01 21:34:02.569547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.569517 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:34:02.569547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.569518 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:34:02.569786 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.569571 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:34:02.572046 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.572022 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.572112 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.572061 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68w5\" (UniqueName: \"kubernetes.io/projected/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-kube-api-access-r68w5\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.572369 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.572313 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.573235 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.573214 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:34:02.673511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.673472 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.673511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.673519 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68w5\" (UniqueName: \"kubernetes.io/projected/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-kube-api-access-r68w5\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.673815 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.673617 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.674180 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.674155 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.676128 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.676108 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.730198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.730122 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68w5\" (UniqueName: \"kubernetes.io/projected/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-kube-api-access-r68w5\") pod \"maas-api-revoke-sub-ec843018-rnpw4\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.784663 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.784622 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:02.871963 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:02.871928 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:34:03.157679 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:03.157649 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4"] Nov 01 21:34:03.159170 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:34:03.159144 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80b6b6e_a5b0_40ed_8dd8_98f600d76cc3.slice/crio-fb7ce145dcaeafeb7cd5235d6696a131807f71cbc0f64fddf31f6c0ada09f547 WatchSource:0}: Error finding container fb7ce145dcaeafeb7cd5235d6696a131807f71cbc0f64fddf31f6c0ada09f547: Status 404 returned error can't find the container with id fb7ce145dcaeafeb7cd5235d6696a131807f71cbc0f64fddf31f6c0ada09f547 Nov 01 21:34:04.073679 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:04.073645 5633 generic.go:356] "Generic (PLEG): container finished" podID="c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" containerID="16432dc2a55af36b477f2e13b5a869771fe810494d9fa011a0d3fcdc46e8e4ba" exitCode=0 Nov 01 21:34:04.074070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:04.073727 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" event={"ID":"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3","Type":"ContainerDied","Data":"16432dc2a55af36b477f2e13b5a869771fe810494d9fa011a0d3fcdc46e8e4ba"} Nov 01 21:34:04.074070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:04.073789 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" event={"ID":"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3","Type":"ContainerStarted","Data":"fb7ce145dcaeafeb7cd5235d6696a131807f71cbc0f64fddf31f6c0ada09f547"} Nov 01 21:34:05.235880 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.235852 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:05.401379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.401261 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-openshift-ca\") pod \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " Nov 01 21:34:05.401379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.401321 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-kube-api-access-r68w5\" (UniqueName: \"kubernetes.io/projected/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-kube-api-access-r68w5\") pod \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " Nov 01 21:34:05.401633 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.401419 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-xks-ca\") pod \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\" (UID: \"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3\") " Nov 01 21:34:05.401741 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.401713 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-openshift-ca" pod "c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" (UID: "c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:34:05.401946 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.401927 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:05.403724 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.403691 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-xks-ca" pod "c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" (UID: "c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:34:05.403804 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.403747 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-kube-api-access-r68w5" pod "c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" (UID: "c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3"). InnerVolumeSpecName "kube-api-access-r68w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:34:05.502786 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.502738 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r68w5\" (UniqueName: \"kubernetes.io/projected/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-kube-api-access-r68w5\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:05.502786 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:05.502783 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:06.086117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:06.086085 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" Nov 01 21:34:06.086117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:06.086094 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4" event={"ID":"c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3","Type":"ContainerDied","Data":"fb7ce145dcaeafeb7cd5235d6696a131807f71cbc0f64fddf31f6c0ada09f547"} Nov 01 21:34:06.086364 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:06.086146 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb7ce145dcaeafeb7cd5235d6696a131807f71cbc0f64fddf31f6c0ada09f547" Nov 01 21:34:16.157396 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:34:16.157352 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:34:20.501297 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:20.501260 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:21.374816 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:21.374771 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:34:36.047839 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:36.047800 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s"] Nov 01 21:34:36.096069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:36.096034 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00d62574-6pf5s"] Nov 01 21:34:36.465851 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:36.465814 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be974d7c-410d-4d34-a33e-3228354bbe8c" path="/var/lib/kubelet/pods/be974d7c-410d-4d34-a33e-3228354bbe8c/volumes" Nov 01 21:34:40.539096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.539055 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc"] Nov 01 21:34:40.539534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.539520 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" containerName="revoke-keys" Nov 01 21:34:40.539581 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.539535 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" containerName="revoke-keys" Nov 01 21:34:40.539643 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.539634 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" containerName="revoke-keys" Nov 01 21:34:40.543105 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.543088 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.599742 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.599702 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc"] Nov 01 21:34:40.634276 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.634239 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwq4\" (UniqueName: \"kubernetes.io/projected/124f961a-eeb5-42f4-af8b-2de865eef12f-kube-api-access-vfwq4\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.634485 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.634295 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.634485 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.634452 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.644965 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.644927 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:34:40.645120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.644974 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:34:40.645120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.644993 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:34:40.645120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.645075 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:34:40.735154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.735096 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.735380 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.735233 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwq4\" (UniqueName: \"kubernetes.io/projected/124f961a-eeb5-42f4-af8b-2de865eef12f-kube-api-access-vfwq4\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.735380 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.735281 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.735984 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.735965 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.737756 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.737733 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.791506 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.791424 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwq4\" (UniqueName: \"kubernetes.io/projected/124f961a-eeb5-42f4-af8b-2de865eef12f-kube-api-access-vfwq4\") pod \"maas-api-revoke-sub-046e1343-9mcbc\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:40.856244 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:40.856209 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:41.228038 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:41.228005 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc"] Nov 01 21:34:41.229626 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:34:41.229600 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124f961a_eeb5_42f4_af8b_2de865eef12f.slice/crio-cd29be927ea8c5f2ce5c98a258e57ad7818395dcadc251e61b10ff76c6fd2bca WatchSource:0}: Error finding container cd29be927ea8c5f2ce5c98a258e57ad7818395dcadc251e61b10ff76c6fd2bca: Status 404 returned error can't find the container with id cd29be927ea8c5f2ce5c98a258e57ad7818395dcadc251e61b10ff76c6fd2bca Nov 01 21:34:41.341733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:41.341696 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" event={"ID":"124f961a-eeb5-42f4-af8b-2de865eef12f","Type":"ContainerStarted","Data":"cd29be927ea8c5f2ce5c98a258e57ad7818395dcadc251e61b10ff76c6fd2bca"} Nov 01 21:34:42.349117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:42.349033 5633 generic.go:356] "Generic (PLEG): container finished" podID="124f961a-eeb5-42f4-af8b-2de865eef12f" containerID="010064b36ec2e12fe1daa8a1f3960ede6c9f47dd31b3d7164c36878e01c8d4f1" exitCode=0 Nov 01 21:34:42.349117 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:42.349095 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" event={"ID":"124f961a-eeb5-42f4-af8b-2de865eef12f","Type":"ContainerDied","Data":"010064b36ec2e12fe1daa8a1f3960ede6c9f47dd31b3d7164c36878e01c8d4f1"} Nov 01 21:34:43.500009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.499982 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:43.563667 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.563637 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-xks-ca\") pod \"124f961a-eeb5-42f4-af8b-2de865eef12f\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " Nov 01 21:34:43.563863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.563783 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-openshift-ca\") pod \"124f961a-eeb5-42f4-af8b-2de865eef12f\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " Nov 01 21:34:43.563863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.563830 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124f961a-eeb5-42f4-af8b-2de865eef12f-kube-api-access-vfwq4\" (UniqueName: \"kubernetes.io/projected/124f961a-eeb5-42f4-af8b-2de865eef12f-kube-api-access-vfwq4\") pod \"124f961a-eeb5-42f4-af8b-2de865eef12f\" (UID: \"124f961a-eeb5-42f4-af8b-2de865eef12f\") " Nov 01 21:34:43.564206 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.564176 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-openshift-ca" pod "124f961a-eeb5-42f4-af8b-2de865eef12f" (UID: "124f961a-eeb5-42f4-af8b-2de865eef12f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:34:43.565870 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.565847 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-xks-ca" pod "124f961a-eeb5-42f4-af8b-2de865eef12f" (UID: "124f961a-eeb5-42f4-af8b-2de865eef12f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:34:43.566011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.565994 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124f961a-eeb5-42f4-af8b-2de865eef12f-kube-api-access-vfwq4" pod "124f961a-eeb5-42f4-af8b-2de865eef12f" (UID: "124f961a-eeb5-42f4-af8b-2de865eef12f"). InnerVolumeSpecName "kube-api-access-vfwq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:34:43.665774 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.665689 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:43.665774 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.665721 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwq4\" (UniqueName: \"kubernetes.io/projected/124f961a-eeb5-42f4-af8b-2de865eef12f-kube-api-access-vfwq4\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:43.665774 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:43.665731 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124f961a-eeb5-42f4-af8b-2de865eef12f-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:44.361540 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:44.361510 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" Nov 01 21:34:44.361764 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:44.361540 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc" event={"ID":"124f961a-eeb5-42f4-af8b-2de865eef12f","Type":"ContainerDied","Data":"cd29be927ea8c5f2ce5c98a258e57ad7818395dcadc251e61b10ff76c6fd2bca"} Nov 01 21:34:44.361764 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:44.361592 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd29be927ea8c5f2ce5c98a258e57ad7818395dcadc251e61b10ff76c6fd2bca" Nov 01 21:34:46.157238 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:34:46.157200 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:34:48.896121 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:48.896088 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:34:51.301919 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.301836 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf"] Nov 01 21:34:51.302312 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.302298 5633 cpu_manager.go:414] "containerMap: removing container" podUID="124f961a-eeb5-42f4-af8b-2de865eef12f" containerName="revoke-keys" Nov 01 21:34:51.302404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.302313 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124f961a-eeb5-42f4-af8b-2de865eef12f" containerName="revoke-keys" Nov 01 21:34:51.302446 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.302441 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="124f961a-eeb5-42f4-af8b-2de865eef12f" containerName="revoke-keys" Nov 01 21:34:51.305846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.305822 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.358278 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.358243 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:34:51.361212 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.361181 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf"] Nov 01 21:34:51.406676 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.406638 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:34:51.406676 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.406659 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:34:51.406676 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.406659 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:34:51.447666 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.447631 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.447834 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.447781 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.447879 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.447848 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42565\" (UniqueName: \"kubernetes.io/projected/c55342f3-1164-4484-ad10-73dcda05bc79-kube-api-access-42565\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.549434 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.549393 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.549663 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.549494 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.549663 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.549540 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42565\" (UniqueName: \"kubernetes.io/projected/c55342f3-1164-4484-ad10-73dcda05bc79-kube-api-access-42565\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.550046 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.550022 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.552099 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.552049 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.606384 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.606347 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42565\" (UniqueName: \"kubernetes.io/projected/c55342f3-1164-4484-ad10-73dcda05bc79-kube-api-access-42565\") pod \"maas-api-revoke-sub-d3fba056-8npvf\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.620201 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.620174 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:51.995214 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:51.995185 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf"] Nov 01 21:34:51.996779 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:34:51.996751 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55342f3_1164_4484_ad10_73dcda05bc79.slice/crio-d1ccd23e9106b8278973f9a225df45fd2de1c8df5882adff8b073b0d4b355fa5 WatchSource:0}: Error finding container d1ccd23e9106b8278973f9a225df45fd2de1c8df5882adff8b073b0d4b355fa5: Status 404 returned error can't find the container with id d1ccd23e9106b8278973f9a225df45fd2de1c8df5882adff8b073b0d4b355fa5 Nov 01 21:34:52.410421 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:52.410375 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" event={"ID":"c55342f3-1164-4484-ad10-73dcda05bc79","Type":"ContainerStarted","Data":"743cbc2fa9457512b5c926c1b76b730ce962479ce172d073b959e2036d0b214f"} Nov 01 21:34:52.410837 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:52.410442 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" event={"ID":"c55342f3-1164-4484-ad10-73dcda05bc79","Type":"ContainerStarted","Data":"d1ccd23e9106b8278973f9a225df45fd2de1c8df5882adff8b073b0d4b355fa5"} Nov 01 21:34:52.526748 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:52.526680 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" podStartSLOduration=1.5266599090000001 podStartE2EDuration="1.526659909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:34:52.52301918 +0000 UTC m=+5500.197278435" watchObservedRunningTime="2026-11-01 21:34:52.526659909 +0000 UTC m=+5500.200919165" Nov 01 21:34:53.394472 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:53.394430 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:34:53.417612 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:53.417573 5633 generic.go:356] "Generic (PLEG): container finished" podID="c55342f3-1164-4484-ad10-73dcda05bc79" containerID="743cbc2fa9457512b5c926c1b76b730ce962479ce172d073b959e2036d0b214f" exitCode=0 Nov 01 21:34:53.418031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:53.417642 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" event={"ID":"c55342f3-1164-4484-ad10-73dcda05bc79","Type":"ContainerDied","Data":"743cbc2fa9457512b5c926c1b76b730ce962479ce172d073b959e2036d0b214f"} Nov 01 21:34:54.648894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.648856 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:34:54.785454 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.785414 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-xks-ca\") pod \"c55342f3-1164-4484-ad10-73dcda05bc79\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " Nov 01 21:34:54.785893 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.785864 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-openshift-ca\") pod \"c55342f3-1164-4484-ad10-73dcda05bc79\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " Nov 01 21:34:54.786088 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.786069 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55342f3-1164-4484-ad10-73dcda05bc79-kube-api-access-42565\" (UniqueName: \"kubernetes.io/projected/c55342f3-1164-4484-ad10-73dcda05bc79-kube-api-access-42565\") pod \"c55342f3-1164-4484-ad10-73dcda05bc79\" (UID: \"c55342f3-1164-4484-ad10-73dcda05bc79\") " Nov 01 21:34:54.786543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.786516 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-openshift-ca" pod "c55342f3-1164-4484-ad10-73dcda05bc79" (UID: "c55342f3-1164-4484-ad10-73dcda05bc79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:34:54.786800 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.786776 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:54.790855 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.790804 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-xks-ca" pod "c55342f3-1164-4484-ad10-73dcda05bc79" (UID: "c55342f3-1164-4484-ad10-73dcda05bc79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:34:54.793239 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.793204 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55342f3-1164-4484-ad10-73dcda05bc79-kube-api-access-42565" pod "c55342f3-1164-4484-ad10-73dcda05bc79" (UID: "c55342f3-1164-4484-ad10-73dcda05bc79"). InnerVolumeSpecName "kube-api-access-42565". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:34:54.888731 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.888681 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42565\" (UniqueName: \"kubernetes.io/projected/c55342f3-1164-4484-ad10-73dcda05bc79-kube-api-access-42565\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:54.888731 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:54.888730 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55342f3-1164-4484-ad10-73dcda05bc79-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:34:55.462358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:55.462058 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" event={"ID":"c55342f3-1164-4484-ad10-73dcda05bc79","Type":"ContainerDied","Data":"d1ccd23e9106b8278973f9a225df45fd2de1c8df5882adff8b073b0d4b355fa5"} Nov 01 21:34:55.462358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:55.462108 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1ccd23e9106b8278973f9a225df45fd2de1c8df5882adff8b073b0d4b355fa5" Nov 01 21:34:55.462358 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:34:55.462154 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf" Nov 01 21:35:03.988195 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:03.988157 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:35:15.187723 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:15.187682 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:35:16.155925 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:35:16.155888 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:35:25.837252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.837213 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk"] Nov 01 21:35:25.837778 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.837738 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c55342f3-1164-4484-ad10-73dcda05bc79" containerName="revoke-keys" Nov 01 21:35:25.837778 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.837750 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55342f3-1164-4484-ad10-73dcda05bc79" containerName="revoke-keys" Nov 01 21:35:25.837855 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.837818 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55342f3-1164-4484-ad10-73dcda05bc79" containerName="revoke-keys" Nov 01 21:35:25.842310 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.842292 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:25.899086 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.899048 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk"] Nov 01 21:35:25.941773 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.941738 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:35:25.941773 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.941746 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:35:25.941773 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.941752 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:35:25.943456 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:25.943435 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:35:26.014342 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.014304 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.014525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.014372 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkhp\" (UniqueName: \"kubernetes.io/projected/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-kube-api-access-6jkhp\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.014525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.014449 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.116109 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.116019 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.116109 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.116058 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkhp\" (UniqueName: \"kubernetes.io/projected/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-kube-api-access-6jkhp\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.116109 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.116080 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.116882 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.116852 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.118656 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.118629 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.172097 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.172067 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkhp\" (UniqueName: \"kubernetes.io/projected/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-kube-api-access-6jkhp\") pod \"maas-api-revoke-sub-ae594979-bn2sk\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.456690 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.456590 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:26.832002 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:26.831968 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk"] Nov 01 21:35:26.832937 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:35:26.832906 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb490cb3_6f96_43cc_aaf5_bca6bc500ce1.slice/crio-83f07e2be696ec43be08672dc595f8bc8a5e7c0dca1ac2117b62bece024db39c WatchSource:0}: Error finding container 83f07e2be696ec43be08672dc595f8bc8a5e7c0dca1ac2117b62bece024db39c: Status 404 returned error can't find the container with id 83f07e2be696ec43be08672dc595f8bc8a5e7c0dca1ac2117b62bece024db39c Nov 01 21:35:27.680504 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:27.680468 5633 generic.go:356] "Generic (PLEG): container finished" podID="db490cb3-6f96-43cc-aaf5-bca6bc500ce1" containerID="3cc4ea5f02d3d56eb34740979567568ce7af38d6d4df6bd05346d921a2d51387" exitCode=0 Nov 01 21:35:27.680912 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:27.680540 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" event={"ID":"db490cb3-6f96-43cc-aaf5-bca6bc500ce1","Type":"ContainerDied","Data":"3cc4ea5f02d3d56eb34740979567568ce7af38d6d4df6bd05346d921a2d51387"} Nov 01 21:35:27.680912 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:27.680583 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" event={"ID":"db490cb3-6f96-43cc-aaf5-bca6bc500ce1","Type":"ContainerStarted","Data":"83f07e2be696ec43be08672dc595f8bc8a5e7c0dca1ac2117b62bece024db39c"} Nov 01 21:35:28.830096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:28.830061 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:28.948895 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:28.948808 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-openshift-ca\") pod \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " Nov 01 21:35:28.948895 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:28.948865 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-xks-ca\") pod \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " Nov 01 21:35:28.949154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:28.948950 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-kube-api-access-6jkhp\" (UniqueName: \"kubernetes.io/projected/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-kube-api-access-6jkhp\") pod \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\" (UID: \"db490cb3-6f96-43cc-aaf5-bca6bc500ce1\") " Nov 01 21:35:28.949282 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:28.949252 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-openshift-ca" pod "db490cb3-6f96-43cc-aaf5-bca6bc500ce1" (UID: "db490cb3-6f96-43cc-aaf5-bca6bc500ce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:35:28.951592 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:28.951556 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-kube-api-access-6jkhp" pod "db490cb3-6f96-43cc-aaf5-bca6bc500ce1" (UID: "db490cb3-6f96-43cc-aaf5-bca6bc500ce1"). InnerVolumeSpecName "kube-api-access-6jkhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:35:28.951733 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:28.951632 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-xks-ca" pod "db490cb3-6f96-43cc-aaf5-bca6bc500ce1" (UID: "db490cb3-6f96-43cc-aaf5-bca6bc500ce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:35:29.050744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:29.050706 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:29.050744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:29.050736 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:29.050744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:29.050746 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jkhp\" (UniqueName: \"kubernetes.io/projected/db490cb3-6f96-43cc-aaf5-bca6bc500ce1-kube-api-access-6jkhp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:29.693654 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:29.693621 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" event={"ID":"db490cb3-6f96-43cc-aaf5-bca6bc500ce1","Type":"ContainerDied","Data":"83f07e2be696ec43be08672dc595f8bc8a5e7c0dca1ac2117b62bece024db39c"} Nov 01 21:35:29.693654 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:29.693647 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk" Nov 01 21:35:29.693886 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:29.693673 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f07e2be696ec43be08672dc595f8bc8a5e7c0dca1ac2117b62bece024db39c" Nov 01 21:35:38.661118 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:38.661074 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:35:46.184093 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:35:46.184053 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:35:47.352520 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:47.352477 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:35:47.353243 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:47.353206 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:35:47.448518 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:47.448471 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:35:47.448863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:47.448833 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:35:48.400234 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.400195 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:35:48.575674 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.575641 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t"] Nov 01 21:35:48.576114 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.576102 5633 cpu_manager.go:414] "containerMap: removing container" podUID="db490cb3-6f96-43cc-aaf5-bca6bc500ce1" containerName="revoke-keys" Nov 01 21:35:48.576166 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.576116 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db490cb3-6f96-43cc-aaf5-bca6bc500ce1" containerName="revoke-keys" Nov 01 21:35:48.576207 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.576200 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="db490cb3-6f96-43cc-aaf5-bca6bc500ce1" containerName="revoke-keys" Nov 01 21:35:48.579913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.579890 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.635203 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.635162 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t"] Nov 01 21:35:48.679990 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.679905 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:35:48.679990 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.679932 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:35:48.681435 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.681412 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:35:48.681560 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.681445 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:35:48.691885 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.691858 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nk2r\" (UniqueName: \"kubernetes.io/projected/9d5672a6-db38-4765-b9c1-d39823a68a0a-kube-api-access-2nk2r\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.692055 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.692006 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.692106 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.692072 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.792989 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.792952 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nk2r\" (UniqueName: \"kubernetes.io/projected/9d5672a6-db38-4765-b9c1-d39823a68a0a-kube-api-access-2nk2r\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.793176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.793045 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.793176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.793081 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.793752 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.793718 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.795639 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.795616 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.849532 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.849495 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nk2r\" (UniqueName: \"kubernetes.io/projected/9d5672a6-db38-4765-b9c1-d39823a68a0a-kube-api-access-2nk2r\") pod \"maas-api-revoke-sub-a4aef7d7-4n54t\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:48.898708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:48.898672 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:49.221842 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:49.221804 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:35:49.269967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:49.269939 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t"] Nov 01 21:35:49.270964 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:35:49.270933 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5672a6_db38_4765_b9c1_d39823a68a0a.slice/crio-0cd0b5e4c980e6eeae3df39ba248290b909bad4c37c8b49c83e0a5fe8a8431ae WatchSource:0}: Error finding container 0cd0b5e4c980e6eeae3df39ba248290b909bad4c37c8b49c83e0a5fe8a8431ae: Status 404 returned error can't find the container with id 0cd0b5e4c980e6eeae3df39ba248290b909bad4c37c8b49c83e0a5fe8a8431ae Nov 01 21:35:49.837162 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:49.837062 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" event={"ID":"9d5672a6-db38-4765-b9c1-d39823a68a0a","Type":"ContainerStarted","Data":"108b08ba38def795a157c785e9101077b14e56ebf3bbe7396252cdd7ced5f179"} Nov 01 21:35:49.837162 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:49.837121 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" event={"ID":"9d5672a6-db38-4765-b9c1-d39823a68a0a","Type":"ContainerStarted","Data":"0cd0b5e4c980e6eeae3df39ba248290b909bad4c37c8b49c83e0a5fe8a8431ae"} Nov 01 21:35:49.950735 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:49.950684 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" podStartSLOduration=1.950669582 podStartE2EDuration="1.950669582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:35:49.948396541 +0000 UTC m=+5557.622655797" watchObservedRunningTime="2026-11-01 21:35:49.950669582 +0000 UTC m=+5557.624928837" Nov 01 21:35:50.843584 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:50.843550 5633 generic.go:356] "Generic (PLEG): container finished" podID="9d5672a6-db38-4765-b9c1-d39823a68a0a" containerID="108b08ba38def795a157c785e9101077b14e56ebf3bbe7396252cdd7ced5f179" exitCode=0 Nov 01 21:35:50.843956 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:50.843629 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" event={"ID":"9d5672a6-db38-4765-b9c1-d39823a68a0a","Type":"ContainerDied","Data":"108b08ba38def795a157c785e9101077b14e56ebf3bbe7396252cdd7ced5f179"} Nov 01 21:35:51.994203 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:51.994176 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:52.024923 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.024880 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5672a6-db38-4765-b9c1-d39823a68a0a-kube-api-access-2nk2r\" (UniqueName: \"kubernetes.io/projected/9d5672a6-db38-4765-b9c1-d39823a68a0a-kube-api-access-2nk2r\") pod \"9d5672a6-db38-4765-b9c1-d39823a68a0a\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " Nov 01 21:35:52.025132 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.024979 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-xks-ca\") pod \"9d5672a6-db38-4765-b9c1-d39823a68a0a\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " Nov 01 21:35:52.025132 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.025018 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-openshift-ca\") pod \"9d5672a6-db38-4765-b9c1-d39823a68a0a\" (UID: \"9d5672a6-db38-4765-b9c1-d39823a68a0a\") " Nov 01 21:35:52.025479 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.025452 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-openshift-ca" pod "9d5672a6-db38-4765-b9c1-d39823a68a0a" (UID: "9d5672a6-db38-4765-b9c1-d39823a68a0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:35:52.027271 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.027240 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-xks-ca" pod "9d5672a6-db38-4765-b9c1-d39823a68a0a" (UID: "9d5672a6-db38-4765-b9c1-d39823a68a0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:35:52.027271 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.027249 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5672a6-db38-4765-b9c1-d39823a68a0a-kube-api-access-2nk2r" pod "9d5672a6-db38-4765-b9c1-d39823a68a0a" (UID: "9d5672a6-db38-4765-b9c1-d39823a68a0a"). InnerVolumeSpecName "kube-api-access-2nk2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:35:52.126805 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.126716 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nk2r\" (UniqueName: \"kubernetes.io/projected/9d5672a6-db38-4765-b9c1-d39823a68a0a-kube-api-access-2nk2r\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:52.126805 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.126746 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:52.126805 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.126756 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5672a6-db38-4765-b9c1-d39823a68a0a-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:35:52.855692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.855614 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" Nov 01 21:35:52.855853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.855613 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t" event={"ID":"9d5672a6-db38-4765-b9c1-d39823a68a0a","Type":"ContainerDied","Data":"0cd0b5e4c980e6eeae3df39ba248290b909bad4c37c8b49c83e0a5fe8a8431ae"} Nov 01 21:35:52.855853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:52.855736 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd0b5e4c980e6eeae3df39ba248290b909bad4c37c8b49c83e0a5fe8a8431ae" Nov 01 21:35:56.764429 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:35:56.764382 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:36:06.594265 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:06.594227 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:36:09.508359 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.508044 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq"] Nov 01 21:36:09.508903 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.508760 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9d5672a6-db38-4765-b9c1-d39823a68a0a" containerName="revoke-keys" Nov 01 21:36:09.508903 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.508780 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5672a6-db38-4765-b9c1-d39823a68a0a" containerName="revoke-keys" Nov 01 21:36:09.509041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.508935 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5672a6-db38-4765-b9c1-d39823a68a0a" containerName="revoke-keys" Nov 01 21:36:09.531947 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.531911 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.573204 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.573160 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq"] Nov 01 21:36:09.584827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.584802 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:36:09.612711 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.612673 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:36:09.613145 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.613121 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:36:09.613399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.613381 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:36:09.648359 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.644992 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcgp\" (UniqueName: \"kubernetes.io/projected/a69ff16c-f13a-401d-9831-5e1100307c00-kube-api-access-2gcgp\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.648359 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.645132 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.648359 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.645200 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.745950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.745902 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.754478 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.754436 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcgp\" (UniqueName: \"kubernetes.io/projected/a69ff16c-f13a-401d-9831-5e1100307c00-kube-api-access-2gcgp\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.754930 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.754907 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.755600 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.750186 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.763191 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.763100 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.834189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.834102 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcgp\" (UniqueName: \"kubernetes.io/projected/a69ff16c-f13a-401d-9831-5e1100307c00-kube-api-access-2gcgp\") pod \"maas-api-revoke-sub-79805ca9-fl6lq\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:09.931556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:09.931515 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:10.438418 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:10.434467 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq"] Nov 01 21:36:10.466414 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:36:10.466378 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69ff16c_f13a_401d_9831_5e1100307c00.slice/crio-28b06966be660a0cd7f537bc83bd40c2a11fc6bff41faf794fcf5a035dbc9d48 WatchSource:0}: Error finding container 28b06966be660a0cd7f537bc83bd40c2a11fc6bff41faf794fcf5a035dbc9d48: Status 404 returned error can't find the container with id 28b06966be660a0cd7f537bc83bd40c2a11fc6bff41faf794fcf5a035dbc9d48 Nov 01 21:36:10.983205 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:10.983165 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" event={"ID":"a69ff16c-f13a-401d-9831-5e1100307c00","Type":"ContainerStarted","Data":"3442604bddd06b367e67b566078e41fcbb59f092bd1e02572731fad926786e34"} Nov 01 21:36:10.984483 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:10.984455 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" event={"ID":"a69ff16c-f13a-401d-9831-5e1100307c00","Type":"ContainerStarted","Data":"28b06966be660a0cd7f537bc83bd40c2a11fc6bff41faf794fcf5a035dbc9d48"} Nov 01 21:36:11.096039 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:11.095984 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" podStartSLOduration=2.095969386 podStartE2EDuration="2.095969386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:36:11.094930028 +0000 UTC m=+5578.769189284" watchObservedRunningTime="2026-11-01 21:36:11.095969386 +0000 UTC m=+5578.770228637" Nov 01 21:36:11.991664 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:11.991629 5633 generic.go:356] "Generic (PLEG): container finished" podID="a69ff16c-f13a-401d-9831-5e1100307c00" containerID="3442604bddd06b367e67b566078e41fcbb59f092bd1e02572731fad926786e34" exitCode=0 Nov 01 21:36:11.992131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:11.991685 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" event={"ID":"a69ff16c-f13a-401d-9831-5e1100307c00","Type":"ContainerDied","Data":"3442604bddd06b367e67b566078e41fcbb59f092bd1e02572731fad926786e34"} Nov 01 21:36:13.137628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.137602 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:13.202127 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.202092 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-xks-ca\") pod \"a69ff16c-f13a-401d-9831-5e1100307c00\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " Nov 01 21:36:13.202399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.202223 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69ff16c-f13a-401d-9831-5e1100307c00-kube-api-access-2gcgp\" (UniqueName: \"kubernetes.io/projected/a69ff16c-f13a-401d-9831-5e1100307c00-kube-api-access-2gcgp\") pod \"a69ff16c-f13a-401d-9831-5e1100307c00\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " Nov 01 21:36:13.202399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.202267 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-openshift-ca\") pod \"a69ff16c-f13a-401d-9831-5e1100307c00\" (UID: \"a69ff16c-f13a-401d-9831-5e1100307c00\") " Nov 01 21:36:13.202692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.202662 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-openshift-ca" pod "a69ff16c-f13a-401d-9831-5e1100307c00" (UID: "a69ff16c-f13a-401d-9831-5e1100307c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:36:13.204425 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.204392 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-xks-ca" pod "a69ff16c-f13a-401d-9831-5e1100307c00" (UID: "a69ff16c-f13a-401d-9831-5e1100307c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:36:13.204895 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.204869 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69ff16c-f13a-401d-9831-5e1100307c00-kube-api-access-2gcgp" pod "a69ff16c-f13a-401d-9831-5e1100307c00" (UID: "a69ff16c-f13a-401d-9831-5e1100307c00"). InnerVolumeSpecName "kube-api-access-2gcgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:36:13.303611 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.303525 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:13.303611 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.303560 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcgp\" (UniqueName: \"kubernetes.io/projected/a69ff16c-f13a-401d-9831-5e1100307c00-kube-api-access-2gcgp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:13.303611 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:13.303571 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69ff16c-f13a-401d-9831-5e1100307c00-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:14.005584 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:14.005555 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" Nov 01 21:36:14.005822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:14.005595 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq" event={"ID":"a69ff16c-f13a-401d-9831-5e1100307c00","Type":"ContainerDied","Data":"28b06966be660a0cd7f537bc83bd40c2a11fc6bff41faf794fcf5a035dbc9d48"} Nov 01 21:36:14.005822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:14.005645 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28b06966be660a0cd7f537bc83bd40c2a11fc6bff41faf794fcf5a035dbc9d48" Nov 01 21:36:16.156601 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:36:16.156562 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:36:16.194708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:16.194667 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:36:18.884463 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:18.884432 5633 scope.go:122] "RemoveContainer" containerID="2c780fa346b1e7bd836a7dfba6d2ba7f98d74118e002a048aa05cd8b2494a560" Nov 01 21:36:25.172500 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:25.172387 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:36:26.863026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.862984 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg"] Nov 01 21:36:26.864444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.864392 5633 cpu_manager.go:414] "containerMap: removing container" podUID="a69ff16c-f13a-401d-9831-5e1100307c00" containerName="revoke-keys" Nov 01 21:36:26.864444 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.864439 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69ff16c-f13a-401d-9831-5e1100307c00" containerName="revoke-keys" Nov 01 21:36:26.864705 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.864534 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69ff16c-f13a-401d-9831-5e1100307c00" containerName="revoke-keys" Nov 01 21:36:26.869319 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.869296 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:26.942050 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.942011 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg"] Nov 01 21:36:26.959241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.959188 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:26.959663 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.959546 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:26.962376 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.959826 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgk8l\" (UniqueName: \"kubernetes.io/projected/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-kube-api-access-dgk8l\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:26.979787 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.978609 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:36:26.979787 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.978691 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:36:26.979787 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.978756 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:36:26.979787 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:26.978861 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:36:27.075368 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.073307 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:27.075368 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.073580 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:27.079549 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.079495 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgk8l\" (UniqueName: \"kubernetes.io/projected/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-kube-api-access-dgk8l\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:27.082665 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.082629 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:27.161249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.130889 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:27.168093 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.168055 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgk8l\" (UniqueName: \"kubernetes.io/projected/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-kube-api-access-dgk8l\") pod \"maas-api-revoke-sub-0b7e8edb-vjckg\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:27.183528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.183494 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:27.754269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:27.754218 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg"] Nov 01 21:36:28.183459 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:28.183423 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" event={"ID":"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf","Type":"ContainerStarted","Data":"bbf38b7808e5eb171c0fdaffea74535a157f16a64a70eaa5f013164853a2ea0c"} Nov 01 21:36:29.237789 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:29.237730 5633 generic.go:356] "Generic (PLEG): container finished" podID="ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" containerID="1d3861855111ae2e3278c272070630dc3ab2a349056e72935e4026a98cb049f4" exitCode=0 Nov 01 21:36:29.239599 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:29.239451 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" event={"ID":"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf","Type":"ContainerDied","Data":"1d3861855111ae2e3278c272070630dc3ab2a349056e72935e4026a98cb049f4"} Nov 01 21:36:30.385909 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.385882 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:30.463457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.463430 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-xks-ca\") pod \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " Nov 01 21:36:30.463651 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.463502 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-kube-api-access-dgk8l\" (UniqueName: \"kubernetes.io/projected/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-kube-api-access-dgk8l\") pod \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " Nov 01 21:36:30.463651 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.463629 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-openshift-ca\") pod \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\" (UID: \"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf\") " Nov 01 21:36:30.464031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.464000 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-openshift-ca" pod "ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" (UID: "ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:36:30.465963 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.465938 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-kube-api-access-dgk8l" pod "ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" (UID: "ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf"). InnerVolumeSpecName "kube-api-access-dgk8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:36:30.466085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.465938 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-xks-ca" pod "ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" (UID: "ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:36:30.564925 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.564834 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:30.564925 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.564868 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:30.564925 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:30.564882 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgk8l\" (UniqueName: \"kubernetes.io/projected/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf-kube-api-access-dgk8l\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:31.251744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:31.251704 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" event={"ID":"ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf","Type":"ContainerDied","Data":"bbf38b7808e5eb171c0fdaffea74535a157f16a64a70eaa5f013164853a2ea0c"} Nov 01 21:36:31.251932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:31.251763 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbf38b7808e5eb171c0fdaffea74535a157f16a64a70eaa5f013164853a2ea0c" Nov 01 21:36:31.251932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:31.251723 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg" Nov 01 21:36:33.196842 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:33.196806 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:36:38.145178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.145144 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2"] Nov 01 21:36:38.145660 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.145643 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" containerName="revoke-keys" Nov 01 21:36:38.145708 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.145662 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" containerName="revoke-keys" Nov 01 21:36:38.145772 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.145763 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" containerName="revoke-keys" Nov 01 21:36:38.149191 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.149169 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.204551 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.204517 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2"] Nov 01 21:36:38.243505 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.243469 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330676b5-362d-4e80-a147-0852866a34bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.243696 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.243550 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330676b5-362d-4e80-a147-0852866a34bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.243757 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.243681 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75d5k\" (UniqueName: \"kubernetes.io/projected/330676b5-362d-4e80-a147-0852866a34bb-kube-api-access-75d5k\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.248795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.248775 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:36:38.248857 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.248790 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:36:38.248857 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.248817 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:36:38.250530 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.250511 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:36:38.345057 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.345024 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330676b5-362d-4e80-a147-0852866a34bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.345246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.345112 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75d5k\" (UniqueName: \"kubernetes.io/projected/330676b5-362d-4e80-a147-0852866a34bb-kube-api-access-75d5k\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.345246 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.345158 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330676b5-362d-4e80-a147-0852866a34bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.345821 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.345794 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330676b5-362d-4e80-a147-0852866a34bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.347644 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.347619 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330676b5-362d-4e80-a147-0852866a34bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.401199 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.401113 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75d5k\" (UniqueName: \"kubernetes.io/projected/330676b5-362d-4e80-a147-0852866a34bb-kube-api-access-75d5k\") pod \"maas-api-revoke-sub-3d4156f1-nb9h2\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.464738 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.464701 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:38.838140 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:38.838111 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2"] Nov 01 21:36:38.839377 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:36:38.839319 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330676b5_362d_4e80_a147_0852866a34bb.slice/crio-83f6e484c5d1c16244711ada5f52e1456c982a08f7e5a93a97ee96dd8054dc58 WatchSource:0}: Error finding container 83f6e484c5d1c16244711ada5f52e1456c982a08f7e5a93a97ee96dd8054dc58: Status 404 returned error can't find the container with id 83f6e484c5d1c16244711ada5f52e1456c982a08f7e5a93a97ee96dd8054dc58 Nov 01 21:36:39.309906 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:39.309868 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" event={"ID":"330676b5-362d-4e80-a147-0852866a34bb","Type":"ContainerStarted","Data":"31ee97080156249b901b326178a34e77575f5f8a4757b62d235fc812cb32d791"} Nov 01 21:36:39.310394 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:39.309916 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" event={"ID":"330676b5-362d-4e80-a147-0852866a34bb","Type":"ContainerStarted","Data":"83f6e484c5d1c16244711ada5f52e1456c982a08f7e5a93a97ee96dd8054dc58"} Nov 01 21:36:39.424307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:39.424203 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" podStartSLOduration=1.424188262 podStartE2EDuration="1.424188262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:36:39.421213417 +0000 UTC m=+5607.095472673" watchObservedRunningTime="2026-11-01 21:36:39.424188262 +0000 UTC m=+5607.098447516" Nov 01 21:36:40.317364 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:40.317304 5633 generic.go:356] "Generic (PLEG): container finished" podID="330676b5-362d-4e80-a147-0852866a34bb" containerID="31ee97080156249b901b326178a34e77575f5f8a4757b62d235fc812cb32d791" exitCode=0 Nov 01 21:36:40.317814 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:40.317392 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" event={"ID":"330676b5-362d-4e80-a147-0852866a34bb","Type":"ContainerDied","Data":"31ee97080156249b901b326178a34e77575f5f8a4757b62d235fc812cb32d791"} Nov 01 21:36:41.472995 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.472970 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:41.583547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.583448 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330676b5-362d-4e80-a147-0852866a34bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330676b5-362d-4e80-a147-0852866a34bb-maas-api-xks-ca\") pod \"330676b5-362d-4e80-a147-0852866a34bb\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " Nov 01 21:36:41.583547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.583496 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330676b5-362d-4e80-a147-0852866a34bb-kube-api-access-75d5k\" (UniqueName: \"kubernetes.io/projected/330676b5-362d-4e80-a147-0852866a34bb-kube-api-access-75d5k\") pod \"330676b5-362d-4e80-a147-0852866a34bb\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " Nov 01 21:36:41.584015 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.583562 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330676b5-362d-4e80-a147-0852866a34bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330676b5-362d-4e80-a147-0852866a34bb-maas-api-openshift-ca\") pod \"330676b5-362d-4e80-a147-0852866a34bb\" (UID: \"330676b5-362d-4e80-a147-0852866a34bb\") " Nov 01 21:36:41.584015 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.583917 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330676b5-362d-4e80-a147-0852866a34bb-maas-api-openshift-ca" pod "330676b5-362d-4e80-a147-0852866a34bb" (UID: "330676b5-362d-4e80-a147-0852866a34bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:36:41.584745 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.584718 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330676b5-362d-4e80-a147-0852866a34bb-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:41.586555 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.586528 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330676b5-362d-4e80-a147-0852866a34bb-maas-api-xks-ca" pod "330676b5-362d-4e80-a147-0852866a34bb" (UID: "330676b5-362d-4e80-a147-0852866a34bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:36:41.587259 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.587233 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330676b5-362d-4e80-a147-0852866a34bb-kube-api-access-75d5k" pod "330676b5-362d-4e80-a147-0852866a34bb" (UID: "330676b5-362d-4e80-a147-0852866a34bb"). InnerVolumeSpecName "kube-api-access-75d5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:36:41.685962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.685920 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330676b5-362d-4e80-a147-0852866a34bb-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:41.685962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:41.685957 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75d5k\" (UniqueName: \"kubernetes.io/projected/330676b5-362d-4e80-a147-0852866a34bb-kube-api-access-75d5k\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:36:42.307503 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:42.307460 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:42.331047 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:42.331009 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" event={"ID":"330676b5-362d-4e80-a147-0852866a34bb","Type":"ContainerDied","Data":"83f6e484c5d1c16244711ada5f52e1456c982a08f7e5a93a97ee96dd8054dc58"} Nov 01 21:36:42.331233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:42.331062 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f6e484c5d1c16244711ada5f52e1456c982a08f7e5a93a97ee96dd8054dc58" Nov 01 21:36:42.331233 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:42.331027 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2" Nov 01 21:36:46.156718 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:36:46.156678 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:36:49.534610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:49.534578 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:36:52.665015 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:36:52.664973 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:37:02.347511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:02.347462 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9rk4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:37:02.348176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:02.348137 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:37:02.454547 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:02.454506 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fjrwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:37:02.454865 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:02.454833 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:37:02.754626 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:02.754572 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fvd8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:37:02.755023 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:02.754987 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:37:03.976357 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:03.976288 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:37:05.508822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.508782 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2"] Nov 01 21:37:05.511996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.511967 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nb9h2"] Nov 01 21:37:05.624988 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.624953 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz"] Nov 01 21:37:05.625441 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.625427 5633 cpu_manager.go:414] "containerMap: removing container" podUID="330676b5-362d-4e80-a147-0852866a34bb" containerName="revoke-keys" Nov 01 21:37:05.625497 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.625443 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330676b5-362d-4e80-a147-0852866a34bb" containerName="revoke-keys" Nov 01 21:37:05.625535 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.625527 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="330676b5-362d-4e80-a147-0852866a34bb" containerName="revoke-keys" Nov 01 21:37:05.628861 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.628832 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.681263 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.681221 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:37:05.681263 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.681257 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:37:05.681609 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.681280 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:37:05.681609 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.681352 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:37:05.685309 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.685285 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz"] Nov 01 21:37:05.723351 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.715226 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.723351 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.715296 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxw7\" (UniqueName: \"kubernetes.io/projected/eab0f2fe-2790-4f9e-beb7-1710a474613f-kube-api-access-8vxw7\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.723351 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.715428 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.816372 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.816251 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.816559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.816390 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.816559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.816423 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxw7\" (UniqueName: \"kubernetes.io/projected/eab0f2fe-2790-4f9e-beb7-1710a474613f-kube-api-access-8vxw7\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.817149 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.817065 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.818859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.818835 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.873123 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.873090 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxw7\" (UniqueName: \"kubernetes.io/projected/eab0f2fe-2790-4f9e-beb7-1710a474613f-kube-api-access-8vxw7\") pod \"maas-api-revoke-sub-3d4156f1-p2zrz\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:05.941585 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:05.941535 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:06.317160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:06.317132 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz"] Nov 01 21:37:06.319045 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:37:06.319016 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab0f2fe_2790_4f9e_beb7_1710a474613f.slice/crio-94aa7937d5de9a7418d4fab3ca49aef69dad73cf2486e040c093bf6d4bf85005 WatchSource:0}: Error finding container 94aa7937d5de9a7418d4fab3ca49aef69dad73cf2486e040c093bf6d4bf85005: Status 404 returned error can't find the container with id 94aa7937d5de9a7418d4fab3ca49aef69dad73cf2486e040c093bf6d4bf85005 Nov 01 21:37:06.466121 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:06.466085 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330676b5-362d-4e80-a147-0852866a34bb" path="/var/lib/kubelet/pods/330676b5-362d-4e80-a147-0852866a34bb/volumes" Nov 01 21:37:06.586824 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:06.586736 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" event={"ID":"eab0f2fe-2790-4f9e-beb7-1710a474613f","Type":"ContainerStarted","Data":"94aa7937d5de9a7418d4fab3ca49aef69dad73cf2486e040c093bf6d4bf85005"} Nov 01 21:37:07.594827 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:07.594793 5633 generic.go:356] "Generic (PLEG): container finished" podID="eab0f2fe-2790-4f9e-beb7-1710a474613f" containerID="eb750ed844042c7af5c66e3ca0923191176d09c36d68c9785ba6310cf1874066" exitCode=0 Nov 01 21:37:07.595209 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:07.594869 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" event={"ID":"eab0f2fe-2790-4f9e-beb7-1710a474613f","Type":"ContainerDied","Data":"eb750ed844042c7af5c66e3ca0923191176d09c36d68c9785ba6310cf1874066"} Nov 01 21:37:08.741222 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.741192 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:08.847842 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.847803 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab0f2fe-2790-4f9e-beb7-1710a474613f-kube-api-access-8vxw7\" (UniqueName: \"kubernetes.io/projected/eab0f2fe-2790-4f9e-beb7-1710a474613f-kube-api-access-8vxw7\") pod \"eab0f2fe-2790-4f9e-beb7-1710a474613f\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " Nov 01 21:37:08.848064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.847868 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-openshift-ca\") pod \"eab0f2fe-2790-4f9e-beb7-1710a474613f\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " Nov 01 21:37:08.848064 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.847895 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-xks-ca\") pod \"eab0f2fe-2790-4f9e-beb7-1710a474613f\" (UID: \"eab0f2fe-2790-4f9e-beb7-1710a474613f\") " Nov 01 21:37:08.848271 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.848247 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-openshift-ca" pod "eab0f2fe-2790-4f9e-beb7-1710a474613f" (UID: "eab0f2fe-2790-4f9e-beb7-1710a474613f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:37:08.850362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.850323 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab0f2fe-2790-4f9e-beb7-1710a474613f-kube-api-access-8vxw7" pod "eab0f2fe-2790-4f9e-beb7-1710a474613f" (UID: "eab0f2fe-2790-4f9e-beb7-1710a474613f"). InnerVolumeSpecName "kube-api-access-8vxw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:37:08.850448 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.850323 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-xks-ca" pod "eab0f2fe-2790-4f9e-beb7-1710a474613f" (UID: "eab0f2fe-2790-4f9e-beb7-1710a474613f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:37:08.949254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.949214 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxw7\" (UniqueName: \"kubernetes.io/projected/eab0f2fe-2790-4f9e-beb7-1710a474613f-kube-api-access-8vxw7\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:37:08.949254 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.949251 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:37:08.949578 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:08.949267 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab0f2fe-2790-4f9e-beb7-1710a474613f-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:37:09.608301 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:09.608262 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" Nov 01 21:37:09.608517 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:09.608301 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz" event={"ID":"eab0f2fe-2790-4f9e-beb7-1710a474613f","Type":"ContainerDied","Data":"94aa7937d5de9a7418d4fab3ca49aef69dad73cf2486e040c093bf6d4bf85005"} Nov 01 21:37:09.608517 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:09.608368 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94aa7937d5de9a7418d4fab3ca49aef69dad73cf2486e040c093bf6d4bf85005" Nov 01 21:37:16.156746 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:37:16.156707 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:37:20.202216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:20.202174 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:37:30.204942 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:30.204902 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:37:42.187979 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:42.187941 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:37:46.157621 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:37:46.157579 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:37:49.884039 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:49.883999 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:37:50.507818 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.507779 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw"] Nov 01 21:37:50.508559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.508537 5633 cpu_manager.go:414] "containerMap: removing container" podUID="eab0f2fe-2790-4f9e-beb7-1710a474613f" containerName="revoke-keys" Nov 01 21:37:50.508616 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.508563 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab0f2fe-2790-4f9e-beb7-1710a474613f" containerName="revoke-keys" Nov 01 21:37:50.508760 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.508748 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab0f2fe-2790-4f9e-beb7-1710a474613f" containerName="revoke-keys" Nov 01 21:37:50.512513 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.512488 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.566991 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.566955 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw"] Nov 01 21:37:50.610969 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.610928 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:37:50.610969 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.610935 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:37:50.610969 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.610948 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:37:50.613089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.613067 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:37:50.657994 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.657950 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.658194 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.658081 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.658194 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.658104 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxxn\" (UniqueName: \"kubernetes.io/projected/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-kube-api-access-whxxn\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.759241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.759146 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.759241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.759228 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.759241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.759246 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxxn\" (UniqueName: \"kubernetes.io/projected/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-kube-api-access-whxxn\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.759892 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.759865 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.761763 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.761743 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.815560 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.815526 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxxn\" (UniqueName: \"kubernetes.io/projected/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-kube-api-access-whxxn\") pod \"maas-api-revoke-sub-e2fc9c5b-spdfw\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:50.826457 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:50.826424 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:51.202090 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:51.202058 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw"] Nov 01 21:37:51.202998 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:37:51.202968 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01e4729_eb41_42f9_9b16_e2a10c6cd0ee.slice/crio-99ec58700f824bf8458903927998e7a27450e260f6d1e29ba6fee965eeed2037 WatchSource:0}: Error finding container 99ec58700f824bf8458903927998e7a27450e260f6d1e29ba6fee965eeed2037: Status 404 returned error can't find the container with id 99ec58700f824bf8458903927998e7a27450e260f6d1e29ba6fee965eeed2037 Nov 01 21:37:51.883739 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:51.883645 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" event={"ID":"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee","Type":"ContainerStarted","Data":"ec6286c7a4dd4ba3ef4c40a0342d992c12ca1d489ace2183731e2a7a2944d45d"} Nov 01 21:37:51.883739 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:51.883704 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" event={"ID":"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee","Type":"ContainerStarted","Data":"99ec58700f824bf8458903927998e7a27450e260f6d1e29ba6fee965eeed2037"} Nov 01 21:37:51.996893 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:51.996841 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" podStartSLOduration=1.996826454 podStartE2EDuration="1.996826454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:37:51.995033317 +0000 UTC m=+5679.669292573" watchObservedRunningTime="2026-11-01 21:37:51.996826454 +0000 UTC m=+5679.671085709" Nov 01 21:37:52.891318 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:52.891284 5633 generic.go:356] "Generic (PLEG): container finished" podID="e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" containerID="ec6286c7a4dd4ba3ef4c40a0342d992c12ca1d489ace2183731e2a7a2944d45d" exitCode=0 Nov 01 21:37:52.891820 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:52.891354 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" event={"ID":"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee","Type":"ContainerDied","Data":"ec6286c7a4dd4ba3ef4c40a0342d992c12ca1d489ace2183731e2a7a2944d45d"} Nov 01 21:37:54.042061 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.042032 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:54.086228 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.086194 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-xks-ca\") pod \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " Nov 01 21:37:54.086405 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.086249 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-openshift-ca\") pod \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " Nov 01 21:37:54.086405 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.086355 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-kube-api-access-whxxn\" (UniqueName: \"kubernetes.io/projected/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-kube-api-access-whxxn\") pod \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\" (UID: \"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee\") " Nov 01 21:37:54.086586 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.086565 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-openshift-ca" pod "e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" (UID: "e01e4729-eb41-42f9-9b16-e2a10c6cd0ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:37:54.086755 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.086737 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:37:54.088509 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.088493 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-kube-api-access-whxxn" pod "e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" (UID: "e01e4729-eb41-42f9-9b16-e2a10c6cd0ee"). InnerVolumeSpecName "kube-api-access-whxxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:37:54.088677 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.088658 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-xks-ca" pod "e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" (UID: "e01e4729-eb41-42f9-9b16-e2a10c6cd0ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:37:54.187885 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.187803 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:37:54.187885 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.187831 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxxn\" (UniqueName: \"kubernetes.io/projected/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee-kube-api-access-whxxn\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:37:54.906469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.906371 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" event={"ID":"e01e4729-eb41-42f9-9b16-e2a10c6cd0ee","Type":"ContainerDied","Data":"99ec58700f824bf8458903927998e7a27450e260f6d1e29ba6fee965eeed2037"} Nov 01 21:37:54.906469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.906406 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw" Nov 01 21:37:54.906469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:54.906426 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99ec58700f824bf8458903927998e7a27450e260f6d1e29ba6fee965eeed2037" Nov 01 21:37:57.044892 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:57.044854 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22"] Nov 01 21:37:57.094296 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:57.094261 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-x5l22"] Nov 01 21:37:58.466400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:37:58.466364 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fa631e-256b-4c6e-a83a-36c857b40528" path="/var/lib/kubelet/pods/c3fa631e-256b-4c6e-a83a-36c857b40528/volumes" Nov 01 21:38:01.284349 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.284285 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n"] Nov 01 21:38:01.285002 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.284980 5633 cpu_manager.go:414] "containerMap: removing container" podUID="e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" containerName="revoke-keys" Nov 01 21:38:01.285106 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.285005 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" containerName="revoke-keys" Nov 01 21:38:01.285164 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.285128 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" containerName="revoke-keys" Nov 01 21:38:01.288202 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.288179 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.344291 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.344256 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n"] Nov 01 21:38:01.365853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.365815 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.366027 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.365888 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.366070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.366024 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6782\" (UniqueName: \"kubernetes.io/projected/6820afce-a3c9-4a24-930b-04d83b43a8c2-kube-api-access-h6782\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.388994 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.388952 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:38:01.389198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.389004 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:38:01.389198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.389050 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:38:01.389453 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.389431 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:38:01.467380 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.467341 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6782\" (UniqueName: \"kubernetes.io/projected/6820afce-a3c9-4a24-930b-04d83b43a8c2-kube-api-access-h6782\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.467579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.467423 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.467579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.467481 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.468113 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.468089 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.470033 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.470013 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.523303 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.523269 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6782\" (UniqueName: \"kubernetes.io/projected/6820afce-a3c9-4a24-930b-04d83b43a8c2-kube-api-access-h6782\") pod \"maas-api-revoke-sub-5a29efb3-w9x4n\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.601721 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.601619 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:01.973942 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:01.973912 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n"] Nov 01 21:38:01.975499 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:38:01.975469 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6820afce_a3c9_4a24_930b_04d83b43a8c2.slice/crio-04fc5056abf96047740c4596318a2185aec3df54d12645af56be04bf6094ba27 WatchSource:0}: Error finding container 04fc5056abf96047740c4596318a2185aec3df54d12645af56be04bf6094ba27: Status 404 returned error can't find the container with id 04fc5056abf96047740c4596318a2185aec3df54d12645af56be04bf6094ba27 Nov 01 21:38:02.957538 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:02.957506 5633 generic.go:356] "Generic (PLEG): container finished" podID="6820afce-a3c9-4a24-930b-04d83b43a8c2" containerID="9d836f833d58ae7eae82b325474944b51c02ac1bf75fc28f5589fbfd3cda9c07" exitCode=0 Nov 01 21:38:02.957538 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:02.957544 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" event={"ID":"6820afce-a3c9-4a24-930b-04d83b43a8c2","Type":"ContainerDied","Data":"9d836f833d58ae7eae82b325474944b51c02ac1bf75fc28f5589fbfd3cda9c07"} Nov 01 21:38:02.958002 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:02.957582 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" event={"ID":"6820afce-a3c9-4a24-930b-04d83b43a8c2","Type":"ContainerStarted","Data":"04fc5056abf96047740c4596318a2185aec3df54d12645af56be04bf6094ba27"} Nov 01 21:38:03.379211 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:03.379174 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:38:04.130602 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.130562 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:04.196541 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.196498 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-openshift-ca\") pod \"6820afce-a3c9-4a24-930b-04d83b43a8c2\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " Nov 01 21:38:04.196744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.196598 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6820afce-a3c9-4a24-930b-04d83b43a8c2-kube-api-access-h6782\" (UniqueName: \"kubernetes.io/projected/6820afce-a3c9-4a24-930b-04d83b43a8c2-kube-api-access-h6782\") pod \"6820afce-a3c9-4a24-930b-04d83b43a8c2\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " Nov 01 21:38:04.196744 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.196730 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-xks-ca\") pod \"6820afce-a3c9-4a24-930b-04d83b43a8c2\" (UID: \"6820afce-a3c9-4a24-930b-04d83b43a8c2\") " Nov 01 21:38:04.196999 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.196967 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-openshift-ca" pod "6820afce-a3c9-4a24-930b-04d83b43a8c2" (UID: "6820afce-a3c9-4a24-930b-04d83b43a8c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:38:04.197389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.197196 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:38:04.199628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.199595 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-xks-ca" pod "6820afce-a3c9-4a24-930b-04d83b43a8c2" (UID: "6820afce-a3c9-4a24-930b-04d83b43a8c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:38:04.199755 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.199635 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6820afce-a3c9-4a24-930b-04d83b43a8c2-kube-api-access-h6782" pod "6820afce-a3c9-4a24-930b-04d83b43a8c2" (UID: "6820afce-a3c9-4a24-930b-04d83b43a8c2"). InnerVolumeSpecName "kube-api-access-h6782". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:38:04.298068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.298012 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6820afce-a3c9-4a24-930b-04d83b43a8c2-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:38:04.298068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.298056 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6782\" (UniqueName: \"kubernetes.io/projected/6820afce-a3c9-4a24-930b-04d83b43a8c2-kube-api-access-h6782\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:38:04.974815 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.974786 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" Nov 01 21:38:04.974815 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.974813 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n" event={"ID":"6820afce-a3c9-4a24-930b-04d83b43a8c2","Type":"ContainerDied","Data":"04fc5056abf96047740c4596318a2185aec3df54d12645af56be04bf6094ba27"} Nov 01 21:38:04.975074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:04.974873 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04fc5056abf96047740c4596318a2185aec3df54d12645af56be04bf6094ba27" Nov 01 21:38:13.978124 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:13.978082 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:38:16.156430 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:38:16.156384 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:38:19.747303 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.747260 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz"] Nov 01 21:38:19.796685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.796651 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2zrz"] Nov 01 21:38:19.866512 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.866478 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl"] Nov 01 21:38:19.866976 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.866962 5633 cpu_manager.go:414] "containerMap: removing container" podUID="6820afce-a3c9-4a24-930b-04d83b43a8c2" containerName="revoke-keys" Nov 01 21:38:19.867021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.866978 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6820afce-a3c9-4a24-930b-04d83b43a8c2" containerName="revoke-keys" Nov 01 21:38:19.867063 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.867056 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6820afce-a3c9-4a24-930b-04d83b43a8c2" containerName="revoke-keys" Nov 01 21:38:19.870572 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.870553 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:19.923041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.922992 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:38:19.923241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.923046 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:38:19.923241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.923065 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:38:19.923241 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.923058 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:38:19.926299 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:19.926275 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl"] Nov 01 21:38:20.032367 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.032236 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.032367 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.032274 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tz6\" (UniqueName: \"kubernetes.io/projected/4c2f3c52-4158-4b39-9f0a-994d78990287-kube-api-access-f4tz6\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.032650 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.032437 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.133541 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.133503 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.133541 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.133544 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tz6\" (UniqueName: \"kubernetes.io/projected/4c2f3c52-4158-4b39-9f0a-994d78990287-kube-api-access-f4tz6\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.133828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.133582 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.134268 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.134233 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.136096 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.136077 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.189755 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.189719 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tz6\" (UniqueName: \"kubernetes.io/projected/4c2f3c52-4158-4b39-9f0a-994d78990287-kube-api-access-f4tz6\") pod \"maas-api-revoke-sub-3d4156f1-d5hxl\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.465655 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.465623 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab0f2fe-2790-4f9e-beb7-1710a474613f" path="/var/lib/kubelet/pods/eab0f2fe-2790-4f9e-beb7-1710a474613f/volumes" Nov 01 21:38:20.483752 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.483724 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:20.858138 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:20.858102 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl"] Nov 01 21:38:20.858769 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:38:20.858733 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2f3c52_4158_4b39_9f0a_994d78990287.slice/crio-7bf86faa4f4a681e5c1b7f1add464c0c52f2c1149f044fa3e7c97c3405e220d4 WatchSource:0}: Error finding container 7bf86faa4f4a681e5c1b7f1add464c0c52f2c1149f044fa3e7c97c3405e220d4: Status 404 returned error can't find the container with id 7bf86faa4f4a681e5c1b7f1add464c0c52f2c1149f044fa3e7c97c3405e220d4 Nov 01 21:38:21.077836 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:21.077800 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" event={"ID":"4c2f3c52-4158-4b39-9f0a-994d78990287","Type":"ContainerStarted","Data":"7bf86faa4f4a681e5c1b7f1add464c0c52f2c1149f044fa3e7c97c3405e220d4"} Nov 01 21:38:22.085066 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:22.084980 5633 generic.go:356] "Generic (PLEG): container finished" podID="4c2f3c52-4158-4b39-9f0a-994d78990287" containerID="90591b3430e6958f634751f69b1a3a187f17e0c398e38332b333fba5bb04331c" exitCode=0 Nov 01 21:38:22.085066 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:22.085054 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" event={"ID":"4c2f3c52-4158-4b39-9f0a-994d78990287","Type":"ContainerDied","Data":"90591b3430e6958f634751f69b1a3a187f17e0c398e38332b333fba5bb04331c"} Nov 01 21:38:23.231167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.231138 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:23.368389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.368276 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2f3c52-4158-4b39-9f0a-994d78990287-kube-api-access-f4tz6\" (UniqueName: \"kubernetes.io/projected/4c2f3c52-4158-4b39-9f0a-994d78990287-kube-api-access-f4tz6\") pod \"4c2f3c52-4158-4b39-9f0a-994d78990287\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " Nov 01 21:38:23.368586 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.368428 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-xks-ca\") pod \"4c2f3c52-4158-4b39-9f0a-994d78990287\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " Nov 01 21:38:23.368586 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.368482 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-openshift-ca\") pod \"4c2f3c52-4158-4b39-9f0a-994d78990287\" (UID: \"4c2f3c52-4158-4b39-9f0a-994d78990287\") " Nov 01 21:38:23.369165 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.369110 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-openshift-ca" pod "4c2f3c52-4158-4b39-9f0a-994d78990287" (UID: "4c2f3c52-4158-4b39-9f0a-994d78990287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:38:23.370814 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.370790 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2f3c52-4158-4b39-9f0a-994d78990287-kube-api-access-f4tz6" pod "4c2f3c52-4158-4b39-9f0a-994d78990287" (UID: "4c2f3c52-4158-4b39-9f0a-994d78990287"). InnerVolumeSpecName "kube-api-access-f4tz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:38:23.370948 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.370934 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-xks-ca" pod "4c2f3c52-4158-4b39-9f0a-994d78990287" (UID: "4c2f3c52-4158-4b39-9f0a-994d78990287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:38:23.469685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.469643 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:38:23.469685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.469681 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f3c52-4158-4b39-9f0a-994d78990287-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:38:23.469685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:23.469691 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4tz6\" (UniqueName: \"kubernetes.io/projected/4c2f3c52-4158-4b39-9f0a-994d78990287-kube-api-access-f4tz6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:38:24.098822 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:24.098788 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" Nov 01 21:38:24.099020 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:24.098812 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl" event={"ID":"4c2f3c52-4158-4b39-9f0a-994d78990287","Type":"ContainerDied","Data":"7bf86faa4f4a681e5c1b7f1add464c0c52f2c1149f044fa3e7c97c3405e220d4"} Nov 01 21:38:24.099020 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:24.098875 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf86faa4f4a681e5c1b7f1add464c0c52f2c1149f044fa3e7c97c3405e220d4" Nov 01 21:38:29.044590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:29.044554 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd"] Nov 01 21:38:29.094142 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:29.094109 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-746sd"] Nov 01 21:38:30.466078 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:30.466037 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb62c07d-6972-4deb-918c-542185c30db8" path="/var/lib/kubelet/pods/cb62c07d-6972-4deb-918c-542185c30db8/volumes" Nov 01 21:38:34.390885 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:34.390851 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:38:35.681996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:35.681960 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:36.269481 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:36.269443 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:37.026359 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:37.025533 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:41.693981 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:41.693943 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:41.880824 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:41.880781 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:42.041863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:42.041813 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:44.874173 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:44.874130 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:45.118465 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:45.118430 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:46.156150 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:38:46.156098 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:38:47.740722 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:47.740692 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:47.898967 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:47.898934 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:48.077471 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:48.077374 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:48.377451 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:48.377365 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:38:50.272804 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:50.272761 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:52.568723 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:52.568687 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:38:54.044507 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:54.044464 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv"] Nov 01 21:38:54.093207 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:54.093164 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wq7qv"] Nov 01 21:38:54.467627 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:54.467589 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60288308-2e6f-4621-bac3-0483cbd3aa1f" path="/var/lib/kubelet/pods/60288308-2e6f-4621-bac3-0483cbd3aa1f/volumes" Nov 01 21:38:57.281263 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:38:57.281224 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:00.219237 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:00.219180 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:39:01.979726 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:01.979684 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:06.047420 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:06.047387 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4"] Nov 01 21:39:06.096068 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:06.096030 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rnpw4"] Nov 01 21:39:06.442915 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:06.442819 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:06.465791 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:06.465759 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3" path="/var/lib/kubelet/pods/c80b6b6e-a5b0-40ed-8dd8-98f600d76cc3/volumes" Nov 01 21:39:06.989719 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:06.989684 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:07.127092 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:07.127044 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:07.265782 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:07.265695 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:10.058445 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.058409 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8"] Nov 01 21:39:10.058880 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.058868 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4c2f3c52-4158-4b39-9f0a-994d78990287" containerName="revoke-keys" Nov 01 21:39:10.058927 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.058882 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2f3c52-4158-4b39-9f0a-994d78990287" containerName="revoke-keys" Nov 01 21:39:10.058974 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.058965 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2f3c52-4158-4b39-9f0a-994d78990287" containerName="revoke-keys" Nov 01 21:39:10.069863 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.069827 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.119143 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.119103 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8"] Nov 01 21:39:10.119666 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.119635 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-model-cache\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.120014 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.119987 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-home\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.120131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.120045 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb66172f-db2f-455e-b2b9-7d0599def4d1-tls-certs\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.120131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.120092 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-tmp-dir\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.120131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.120125 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszh6\" (UniqueName: \"kubernetes.io/projected/eb66172f-db2f-455e-b2b9-7d0599def4d1-kube-api-access-rszh6\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.120321 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.120175 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-dshm\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.144078 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.144044 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:10.164220 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.164177 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c660c6\"/\"openshift-service-ca.crt\"" Nov 01 21:39:10.164804 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.164786 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c660c6\"/\"simulator-single-main-c660c6-kserve-self-signed-certs\"" Nov 01 21:39:10.170917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.170893 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c660c6\"/\"kube-root-ca.crt\"" Nov 01 21:39:10.171101 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.171087 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c660c6\"/\"default-dockercfg-6fw9b\"" Nov 01 21:39:10.221623 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.221584 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-home\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.221898 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.221639 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb66172f-db2f-455e-b2b9-7d0599def4d1-tls-certs\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.221898 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.221689 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-tmp-dir\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.222147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.222066 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-home\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.222226 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.222195 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-tmp-dir\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.223033 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.222941 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszh6\" (UniqueName: \"kubernetes.io/projected/eb66172f-db2f-455e-b2b9-7d0599def4d1-kube-api-access-rszh6\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.223177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.223083 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-dshm\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.223229 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.223214 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-model-cache\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.224168 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.224146 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-model-cache\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.225693 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.225668 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb66172f-db2f-455e-b2b9-7d0599def4d1-tls-certs\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.225924 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.225903 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-dshm\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.278669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.278629 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszh6\" (UniqueName: \"kubernetes.io/projected/eb66172f-db2f-455e-b2b9-7d0599def4d1-kube-api-access-rszh6\") pod \"simulator-single-main-c660c6-kserve-55db568b96-4bpg8\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.382308 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.382218 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:10.759517 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:10.759484 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8"] Nov 01 21:39:10.760350 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:39:10.760309 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb66172f_db2f_455e_b2b9_7d0599def4d1.slice/crio-e58ce056fca0a92886bf4ae37299a9009f29cc7e7acf545d186eb6b42e07fac4 WatchSource:0}: Error finding container e58ce056fca0a92886bf4ae37299a9009f29cc7e7acf545d186eb6b42e07fac4: Status 404 returned error can't find the container with id e58ce056fca0a92886bf4ae37299a9009f29cc7e7acf545d186eb6b42e07fac4 Nov 01 21:39:11.382875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:11.382845 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" event={"ID":"eb66172f-db2f-455e-b2b9-7d0599def4d1","Type":"ContainerStarted","Data":"8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c"} Nov 01 21:39:11.383324 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:11.382883 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" event={"ID":"eb66172f-db2f-455e-b2b9-7d0599def4d1","Type":"ContainerStarted","Data":"e58ce056fca0a92886bf4ae37299a9009f29cc7e7acf545d186eb6b42e07fac4"} Nov 01 21:39:11.497623 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:11.497567 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" podStartSLOduration=1.497552037 podStartE2EDuration="1.497552037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:39:11.496527479 +0000 UTC m=+5759.170786737" watchObservedRunningTime="2026-11-01 21:39:11.497552037 +0000 UTC m=+5759.171811288" Nov 01 21:39:11.684474 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:11.684360 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:12.382370 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:12.382321 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:14.685682 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:14.685631 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:39:16.156580 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:39:16.156543 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:39:16.387499 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:16.387463 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:18.931620 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:18.931581 5633 scope.go:122] "RemoveContainer" containerID="04369fde8da0b7a3497bd2695b87f716bd5bdbc172e3309932273cbf67ef8d87" Nov 01 21:39:20.382663 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:20.382621 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:20.396154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:20.396115 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:20.448936 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:20.448900 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:39:21.099875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:21.099792 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:25.779105 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:25.779064 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:30.420636 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:30.420600 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:35.082441 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:35.082405 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:39.775157 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:39.775124 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:40.085543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:40.085455 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:39:44.043555 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:44.043523 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc"] Nov 01 21:39:44.093077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:44.093038 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9mcbc"] Nov 01 21:39:44.465767 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:44.465728 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124f961a-eeb5-42f4-af8b-2de865eef12f" path="/var/lib/kubelet/pods/124f961a-eeb5-42f4-af8b-2de865eef12f/volumes" Nov 01 21:39:44.478533 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:44.478504 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:46.156730 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:39:46.156691 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:39:49.084261 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:49.084206 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:50.566456 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:50.566422 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:53.797670 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:53.797635 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:54.154466 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.154387 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw"] Nov 01 21:39:54.159216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.159182 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.213852 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.213818 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw"] Nov 01 21:39:54.259409 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.259316 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:39:54.259409 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.259403 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:39:54.259625 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.259381 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:39:54.260866 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.260845 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:39:54.305452 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.305416 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.305639 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.305480 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55j9\" (UniqueName: \"kubernetes.io/projected/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-kube-api-access-v55j9\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.305639 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.305578 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.406200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.406111 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.406200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.406164 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55j9\" (UniqueName: \"kubernetes.io/projected/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-kube-api-access-v55j9\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.406200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.406202 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.407084 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.407012 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.408847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.408824 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.462216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.462187 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55j9\" (UniqueName: \"kubernetes.io/projected/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-kube-api-access-v55j9\") pod \"maas-api-revoke-sub-91c95d31-z4mnw\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.476604 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.476567 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:54.859377 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:54.859342 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw"] Nov 01 21:39:54.860481 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:39:54.860448 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37df3fa9_0c07_4091_bdcb_77cb13d05f2b.slice/crio-bd7d944644c85a3dd83c02efa55ba24c4124c45ff7891c47632e8926ff5e4ecd WatchSource:0}: Error finding container bd7d944644c85a3dd83c02efa55ba24c4124c45ff7891c47632e8926ff5e4ecd: Status 404 returned error can't find the container with id bd7d944644c85a3dd83c02efa55ba24c4124c45ff7891c47632e8926ff5e4ecd Nov 01 21:39:55.044685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:55.044650 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf"] Nov 01 21:39:55.048020 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:55.047990 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8npvf"] Nov 01 21:39:55.649558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:55.649513 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" event={"ID":"37df3fa9-0c07-4091-bdcb-77cb13d05f2b","Type":"ContainerStarted","Data":"34fc5951ef02b4d2568ceb940a002fbb35b8d1f83be7bfd6c9b848c213adcc10"} Nov 01 21:39:55.649758 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:55.649576 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" event={"ID":"37df3fa9-0c07-4091-bdcb-77cb13d05f2b","Type":"ContainerStarted","Data":"bd7d944644c85a3dd83c02efa55ba24c4124c45ff7891c47632e8926ff5e4ecd"} Nov 01 21:39:55.763389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:55.763311 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" podStartSLOduration=1.763293634 podStartE2EDuration="1.763293634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:39:55.760856139 +0000 UTC m=+5803.435115394" watchObservedRunningTime="2026-11-01 21:39:55.763293634 +0000 UTC m=+5803.437552910" Nov 01 21:39:56.466800 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:56.466765 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55342f3-1164-4484-ad10-73dcda05bc79" path="/var/lib/kubelet/pods/c55342f3-1164-4484-ad10-73dcda05bc79/volumes" Nov 01 21:39:56.657429 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:56.657393 5633 generic.go:356] "Generic (PLEG): container finished" podID="37df3fa9-0c07-4091-bdcb-77cb13d05f2b" containerID="34fc5951ef02b4d2568ceb940a002fbb35b8d1f83be7bfd6c9b848c213adcc10" exitCode=0 Nov 01 21:39:56.657429 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:56.657420 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" event={"ID":"37df3fa9-0c07-4091-bdcb-77cb13d05f2b","Type":"ContainerDied","Data":"34fc5951ef02b4d2568ceb940a002fbb35b8d1f83be7bfd6c9b848c213adcc10"} Nov 01 21:39:57.808481 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.808448 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:57.947170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.947067 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-openshift-ca\") pod \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " Nov 01 21:39:57.947170 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.947170 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-kube-api-access-v55j9\" (UniqueName: \"kubernetes.io/projected/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-kube-api-access-v55j9\") pod \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " Nov 01 21:39:57.947451 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.947212 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-xks-ca\") pod \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\" (UID: \"37df3fa9-0c07-4091-bdcb-77cb13d05f2b\") " Nov 01 21:39:57.947536 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.947481 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-openshift-ca" pod "37df3fa9-0c07-4091-bdcb-77cb13d05f2b" (UID: "37df3fa9-0c07-4091-bdcb-77cb13d05f2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:39:57.947776 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.947752 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:57.949589 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.949544 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-kube-api-access-v55j9" pod "37df3fa9-0c07-4091-bdcb-77cb13d05f2b" (UID: "37df3fa9-0c07-4091-bdcb-77cb13d05f2b"). InnerVolumeSpecName "kube-api-access-v55j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:39:57.949831 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:57.949805 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-xks-ca" pod "37df3fa9-0c07-4091-bdcb-77cb13d05f2b" (UID: "37df3fa9-0c07-4091-bdcb-77cb13d05f2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:39:58.048947 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:58.048906 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55j9\" (UniqueName: \"kubernetes.io/projected/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-kube-api-access-v55j9\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:58.048947 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:58.048939 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37df3fa9-0c07-4091-bdcb-77cb13d05f2b-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:39:58.490853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:58.490815 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:39:58.670664 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:58.670622 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" event={"ID":"37df3fa9-0c07-4091-bdcb-77cb13d05f2b","Type":"ContainerDied","Data":"bd7d944644c85a3dd83c02efa55ba24c4124c45ff7891c47632e8926ff5e4ecd"} Nov 01 21:39:58.670664 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:58.670648 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw" Nov 01 21:39:58.670913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:39:58.670685 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7d944644c85a3dd83c02efa55ba24c4124c45ff7891c47632e8926ff5e4ecd" Nov 01 21:40:03.187449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:03.187404 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:05.829525 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:05.829485 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:06.241954 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:06.241903 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:07.860165 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:07.860118 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:08.994972 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:08.994934 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:40:10.802174 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:10.802139 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8"] Nov 01 21:40:10.899597 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:10.899555 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:11.885866 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:11.885829 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:12.012990 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:12.012955 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:12.502433 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:12.502398 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:12.668871 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:12.668841 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:16.157689 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:40:16.157650 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:40:17.315840 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:17.315803 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:17.485248 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:17.485208 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:18.957530 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:18.957497 5633 scope.go:122] "RemoveContainer" containerID="23c76b0672e8c9cacb4a6539557bd5d96cdf351e5fff6a56e59c6bab30a77fb5" Nov 01 21:40:18.967765 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:18.967746 5633 scope.go:122] "RemoveContainer" containerID="16432dc2a55af36b477f2e13b5a869771fe810494d9fa011a0d3fcdc46e8e4ba" Nov 01 21:40:18.977342 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:18.977308 5633 scope.go:122] "RemoveContainer" containerID="c9bbde6660187efde2f2a80a7696a9d4eb7aa670d594b10cefd79f697ee1a628" Nov 01 21:40:21.902424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:21.902389 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:40:25.823859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:25.823793 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" podUID="eb66172f-db2f-455e-b2b9-7d0599def4d1" containerName="main" containerID="cri-o://8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c" gracePeriod=45 Nov 01 21:40:26.085449 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.085381 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:40:26.242908 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.242871 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb66172f-db2f-455e-b2b9-7d0599def4d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb66172f-db2f-455e-b2b9-7d0599def4d1-tls-certs\") pod \"eb66172f-db2f-455e-b2b9-7d0599def4d1\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " Nov 01 21:40:26.243089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.242923 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-home\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-home\") pod \"eb66172f-db2f-455e-b2b9-7d0599def4d1\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " Nov 01 21:40:26.243089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.242978 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-dshm\") pod \"eb66172f-db2f-455e-b2b9-7d0599def4d1\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " Nov 01 21:40:26.243089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243018 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-tmp-dir\") pod \"eb66172f-db2f-455e-b2b9-7d0599def4d1\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " Nov 01 21:40:26.243089 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243052 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb66172f-db2f-455e-b2b9-7d0599def4d1-kube-api-access-rszh6\" (UniqueName: \"kubernetes.io/projected/eb66172f-db2f-455e-b2b9-7d0599def4d1-kube-api-access-rszh6\") pod \"eb66172f-db2f-455e-b2b9-7d0599def4d1\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " Nov 01 21:40:26.243309 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243116 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-model-cache\") pod \"eb66172f-db2f-455e-b2b9-7d0599def4d1\" (UID: \"eb66172f-db2f-455e-b2b9-7d0599def4d1\") " Nov 01 21:40:26.243309 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243225 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-home" pod "eb66172f-db2f-455e-b2b9-7d0599def4d1" (UID: "eb66172f-db2f-455e-b2b9-7d0599def4d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:26.243464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243408 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-tmp-dir" pod "eb66172f-db2f-455e-b2b9-7d0599def4d1" (UID: "eb66172f-db2f-455e-b2b9-7d0599def4d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:26.243464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243448 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-model-cache" pod "eb66172f-db2f-455e-b2b9-7d0599def4d1" (UID: "eb66172f-db2f-455e-b2b9-7d0599def4d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:26.243610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243588 5633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-home\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:26.243699 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243638 5633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-tmp-dir\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:26.243768 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.243702 5633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-model-cache\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:26.245487 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.245444 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb66172f-db2f-455e-b2b9-7d0599def4d1-kube-api-access-rszh6" pod "eb66172f-db2f-455e-b2b9-7d0599def4d1" (UID: "eb66172f-db2f-455e-b2b9-7d0599def4d1"). InnerVolumeSpecName "kube-api-access-rszh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:40:26.245607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.245513 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb66172f-db2f-455e-b2b9-7d0599def4d1-tls-certs" pod "eb66172f-db2f-455e-b2b9-7d0599def4d1" (UID: "eb66172f-db2f-455e-b2b9-7d0599def4d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:40:26.245607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.245520 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-dshm" pod "eb66172f-db2f-455e-b2b9-7d0599def4d1" (UID: "eb66172f-db2f-455e-b2b9-7d0599def4d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:40:26.345009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.344901 5633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb66172f-db2f-455e-b2b9-7d0599def4d1-tls-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:26.345009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.344935 5633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb66172f-db2f-455e-b2b9-7d0599def4d1-dshm\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:26.345009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.344949 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszh6\" (UniqueName: \"kubernetes.io/projected/eb66172f-db2f-455e-b2b9-7d0599def4d1-kube-api-access-rszh6\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:40:26.852264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.852226 5633 generic.go:356] "Generic (PLEG): container finished" podID="eb66172f-db2f-455e-b2b9-7d0599def4d1" containerID="8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c" exitCode=0 Nov 01 21:40:26.852799 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.852275 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" Nov 01 21:40:26.852799 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.852313 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" event={"ID":"eb66172f-db2f-455e-b2b9-7d0599def4d1","Type":"ContainerDied","Data":"8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c"} Nov 01 21:40:26.852799 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.852376 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8" event={"ID":"eb66172f-db2f-455e-b2b9-7d0599def4d1","Type":"ContainerDied","Data":"e58ce056fca0a92886bf4ae37299a9009f29cc7e7acf545d186eb6b42e07fac4"} Nov 01 21:40:26.852799 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.852396 5633 scope.go:122] "RemoveContainer" containerID="8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c" Nov 01 21:40:26.862107 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.862086 5633 scope.go:122] "RemoveContainer" containerID="8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c" Nov 01 21:40:26.862425 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:40:26.862405 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c\": container with ID starting with 8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c not found: ID does not exist" containerID="8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c" Nov 01 21:40:26.862490 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:26.862434 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c"} err="failed to get container status \"8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c\": rpc error: code = NotFound desc = could not find container \"8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c\": container with ID starting with 8d4e0246da7a817f5f3d1477dbbbb0a3475e999542076992edfd66fde0eaa66c not found: ID does not exist" Nov 01 21:40:27.014934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:27.014899 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8"] Nov 01 21:40:27.063998 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:27.063960 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c660c6/simulator-single-main-c660c6-kserve-55db568b96-4bpg8"] Nov 01 21:40:28.466465 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:28.466436 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb66172f-db2f-455e-b2b9-7d0599def4d1" path="/var/lib/kubelet/pods/eb66172f-db2f-455e-b2b9-7d0599def4d1/volumes" Nov 01 21:40:29.044221 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:29.044181 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk"] Nov 01 21:40:29.092977 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:29.092944 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bn2sk"] Nov 01 21:40:30.417556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:30.417520 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:30.465817 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:30.465777 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db490cb3-6f96-43cc-aaf5-bca6bc500ce1" path="/var/lib/kubelet/pods/db490cb3-6f96-43cc-aaf5-bca6bc500ce1/volumes" Nov 01 21:40:39.387392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:39.387325 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:46.156143 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:40:46.156100 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:40:46.793929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:46.793892 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:46.967189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:46.967144 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:48.037582 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:48.037546 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:48.168237 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:48.168197 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:49.034544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:49.034511 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:49.275579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:49.275545 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:49.413931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:49.413834 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:49.918472 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:49.918442 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:50.090098 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:50.090062 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:50.929505 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:50.929466 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:52.046452 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:52.046417 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t"] Nov 01 21:40:52.094950 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:52.094915 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4n54t"] Nov 01 21:40:52.467076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:52.467044 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5672a6-db38-4765-b9c1-d39823a68a0a" path="/var/lib/kubelet/pods/9d5672a6-db38-4765-b9c1-d39823a68a0a/volumes" Nov 01 21:40:53.970556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:53.970520 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:54.158564 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:54.158529 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:40:58.680383 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:40:58.680321 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:01.038772 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.038733 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws"] Nov 01 21:41:01.039252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.039227 5633 cpu_manager.go:414] "containerMap: removing container" podUID="eb66172f-db2f-455e-b2b9-7d0599def4d1" containerName="main" Nov 01 21:41:01.039252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.039239 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb66172f-db2f-455e-b2b9-7d0599def4d1" containerName="main" Nov 01 21:41:01.039353 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.039257 5633 cpu_manager.go:414] "containerMap: removing container" podUID="37df3fa9-0c07-4091-bdcb-77cb13d05f2b" containerName="revoke-keys" Nov 01 21:41:01.039353 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.039265 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37df3fa9-0c07-4091-bdcb-77cb13d05f2b" containerName="revoke-keys" Nov 01 21:41:01.039432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.039362 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb66172f-db2f-455e-b2b9-7d0599def4d1" containerName="main" Nov 01 21:41:01.039432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.039373 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="37df3fa9-0c07-4091-bdcb-77cb13d05f2b" containerName="revoke-keys" Nov 01 21:41:01.042267 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.042250 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.055951 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.055917 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:41:01.098962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.098928 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws"] Nov 01 21:41:01.100369 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.100305 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:41:01.100610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.100586 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:41:01.100671 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.100621 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:41:01.100671 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.100634 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:41:01.118307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.118263 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.118539 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.118489 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.118539 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.118528 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frttz\" (UniqueName: \"kubernetes.io/projected/24b271f4-e612-4eb5-a550-5233274ba09a-kube-api-access-frttz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.219774 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.219731 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.219988 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.219776 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frttz\" (UniqueName: \"kubernetes.io/projected/24b271f4-e612-4eb5-a550-5233274ba09a-kube-api-access-frttz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.219988 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.219920 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.220635 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.220609 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.222489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.222468 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.276017 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.275979 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frttz\" (UniqueName: \"kubernetes.io/projected/24b271f4-e612-4eb5-a550-5233274ba09a-kube-api-access-frttz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.358252 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.358169 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:01.731077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:01.731050 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws"] Nov 01 21:41:01.732640 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:41:01.732613 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b271f4_e612_4eb5_a550_5233274ba09a.slice/crio-2d5f86be91ddd7524470e4a1ed0afc0f98da7650b356d08b1ba3423c9cf2bc04 WatchSource:0}: Error finding container 2d5f86be91ddd7524470e4a1ed0afc0f98da7650b356d08b1ba3423c9cf2bc04: Status 404 returned error can't find the container with id 2d5f86be91ddd7524470e4a1ed0afc0f98da7650b356d08b1ba3423c9cf2bc04 Nov 01 21:41:02.073438 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:02.073390 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" event={"ID":"24b271f4-e612-4eb5-a550-5233274ba09a","Type":"ContainerStarted","Data":"116b66a27a95c4ecb4cee4b80ad7a49e427907657f6c6e20cd357b4bb9208748"} Nov 01 21:41:02.073876 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:02.073465 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" event={"ID":"24b271f4-e612-4eb5-a550-5233274ba09a","Type":"ContainerStarted","Data":"2d5f86be91ddd7524470e4a1ed0afc0f98da7650b356d08b1ba3423c9cf2bc04"} Nov 01 21:41:02.186488 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:02.186423 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" podStartSLOduration=2.186406244 podStartE2EDuration="2.186406244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:02.184850252 +0000 UTC m=+5869.859109508" watchObservedRunningTime="2026-11-01 21:41:02.186406244 +0000 UTC m=+5869.860665498" Nov 01 21:41:02.630311 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:02.630268 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:03.081595 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:03.081559 5633 generic.go:356] "Generic (PLEG): container finished" podID="24b271f4-e612-4eb5-a550-5233274ba09a" containerID="116b66a27a95c4ecb4cee4b80ad7a49e427907657f6c6e20cd357b4bb9208748" exitCode=0 Nov 01 21:41:03.082019 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:03.081600 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" event={"ID":"24b271f4-e612-4eb5-a550-5233274ba09a","Type":"ContainerDied","Data":"116b66a27a95c4ecb4cee4b80ad7a49e427907657f6c6e20cd357b4bb9208748"} Nov 01 21:41:03.367453 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:03.367355 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:04.243227 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.243202 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:04.353020 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.352927 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-xks-ca\") pod \"24b271f4-e612-4eb5-a550-5233274ba09a\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " Nov 01 21:41:04.353198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.353057 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b271f4-e612-4eb5-a550-5233274ba09a-kube-api-access-frttz\" (UniqueName: \"kubernetes.io/projected/24b271f4-e612-4eb5-a550-5233274ba09a-kube-api-access-frttz\") pod \"24b271f4-e612-4eb5-a550-5233274ba09a\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " Nov 01 21:41:04.353198 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.353107 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-openshift-ca\") pod \"24b271f4-e612-4eb5-a550-5233274ba09a\" (UID: \"24b271f4-e612-4eb5-a550-5233274ba09a\") " Nov 01 21:41:04.353575 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.353544 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-openshift-ca" pod "24b271f4-e612-4eb5-a550-5233274ba09a" (UID: "24b271f4-e612-4eb5-a550-5233274ba09a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:41:04.355399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.355369 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-xks-ca" pod "24b271f4-e612-4eb5-a550-5233274ba09a" (UID: "24b271f4-e612-4eb5-a550-5233274ba09a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:41:04.355399 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.355390 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b271f4-e612-4eb5-a550-5233274ba09a-kube-api-access-frttz" pod "24b271f4-e612-4eb5-a550-5233274ba09a" (UID: "24b271f4-e612-4eb5-a550-5233274ba09a"). InnerVolumeSpecName "kube-api-access-frttz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:41:04.453929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.453896 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:04.453929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.453926 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b271f4-e612-4eb5-a550-5233274ba09a-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:04.453929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:04.453936 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frttz\" (UniqueName: \"kubernetes.io/projected/24b271f4-e612-4eb5-a550-5233274ba09a-kube-api-access-frttz\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:05.101424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:05.101383 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" event={"ID":"24b271f4-e612-4eb5-a550-5233274ba09a","Type":"ContainerDied","Data":"2d5f86be91ddd7524470e4a1ed0afc0f98da7650b356d08b1ba3423c9cf2bc04"} Nov 01 21:41:05.101424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:05.101409 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws" Nov 01 21:41:05.101660 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:05.101440 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5f86be91ddd7524470e4a1ed0afc0f98da7650b356d08b1ba3423c9cf2bc04" Nov 01 21:41:08.078905 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:08.078863 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:12.767044 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:12.767004 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:14.045958 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:14.045926 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq"] Nov 01 21:41:14.095140 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:14.095105 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fl6lq"] Nov 01 21:41:14.455916 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:14.455829 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:14.466556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:14.466512 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69ff16c-f13a-401d-9831-5e1100307c00" path="/var/lib/kubelet/pods/a69ff16c-f13a-401d-9831-5e1100307c00/volumes" Nov 01 21:41:14.586273 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:14.586236 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:16.157010 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:41:16.156966 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:41:16.894325 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:16.894289 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:18.344721 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:18.344682 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:18.620129 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:18.620038 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:18.950575 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:18.950487 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws"] Nov 01 21:41:18.999315 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:18.999277 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c660c6-9lbws"] Nov 01 21:41:19.011238 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:19.011211 5633 scope.go:122] "RemoveContainer" containerID="010064b36ec2e12fe1daa8a1f3960ede6c9f47dd31b3d7164c36878e01c8d4f1" Nov 01 21:41:19.021147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:19.021125 5633 scope.go:122] "RemoveContainer" containerID="743cbc2fa9457512b5c926c1b76b730ce962479ce172d073b959e2036d0b214f" Nov 01 21:41:19.197799 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:19.197768 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:20.469173 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.469136 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b271f4-e612-4eb5-a550-5233274ba09a" path="/var/lib/kubelet/pods/24b271f4-e612-4eb5-a550-5233274ba09a/volumes" Nov 01 21:41:20.781396 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.781295 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl"] Nov 01 21:41:20.830415 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.830383 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d5hxl"] Nov 01 21:41:20.899167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.899128 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf"] Nov 01 21:41:20.899637 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.899623 5633 cpu_manager.go:414] "containerMap: removing container" podUID="24b271f4-e612-4eb5-a550-5233274ba09a" containerName="revoke-keys" Nov 01 21:41:20.899692 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.899639 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b271f4-e612-4eb5-a550-5233274ba09a" containerName="revoke-keys" Nov 01 21:41:20.899750 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.899740 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b271f4-e612-4eb5-a550-5233274ba09a" containerName="revoke-keys" Nov 01 21:41:20.904859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.904837 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:20.956876 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.956837 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:41:20.957072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.956936 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:41:20.957072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.956936 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:41:20.957335 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.957313 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:41:20.959131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:20.959101 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf"] Nov 01 21:41:21.029874 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.029835 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.029874 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.029877 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.030144 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.029962 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsdl\" (UniqueName: \"kubernetes.io/projected/974fade6-413b-40c0-b6ad-0d206ccbb013-kube-api-access-dcsdl\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.131691 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.131600 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.131691 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.131640 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.131691 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.131668 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsdl\" (UniqueName: \"kubernetes.io/projected/974fade6-413b-40c0-b6ad-0d206ccbb013-kube-api-access-dcsdl\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.132296 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.132273 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.134286 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.134261 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.187687 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.187647 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsdl\" (UniqueName: \"kubernetes.io/projected/974fade6-413b-40c0-b6ad-0d206ccbb013-kube-api-access-dcsdl\") pod \"maas-api-revoke-sub-3d4156f1-brnsf\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.218407 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.218371 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:21.598069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:21.598042 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf"] Nov 01 21:41:21.599101 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:41:21.599076 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974fade6_413b_40c0_b6ad_0d206ccbb013.slice/crio-2747b103c12c817c174c63723fa1bd6920a22c0e15089bf34a20af22f4515e45 WatchSource:0}: Error finding container 2747b103c12c817c174c63723fa1bd6920a22c0e15089bf34a20af22f4515e45: Status 404 returned error can't find the container with id 2747b103c12c817c174c63723fa1bd6920a22c0e15089bf34a20af22f4515e45 Nov 01 21:41:22.207395 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:22.207290 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" event={"ID":"974fade6-413b-40c0-b6ad-0d206ccbb013","Type":"ContainerStarted","Data":"fc4510dbdf17636c2af3263dd01bc102679abd0f3922e0f6d6c0dbf5534c6fff"} Nov 01 21:41:22.207395 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:22.207366 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" event={"ID":"974fade6-413b-40c0-b6ad-0d206ccbb013","Type":"ContainerStarted","Data":"2747b103c12c817c174c63723fa1bd6920a22c0e15089bf34a20af22f4515e45"} Nov 01 21:41:22.318595 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:22.318524 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" podStartSLOduration=2.31828793 podStartE2EDuration="2.31828793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:41:22.317894688 +0000 UTC m=+5889.992153945" watchObservedRunningTime="2026-11-01 21:41:22.31828793 +0000 UTC m=+5889.992547189" Nov 01 21:41:22.466344 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:22.466298 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2f3c52-4158-4b39-9f0a-994d78990287" path="/var/lib/kubelet/pods/4c2f3c52-4158-4b39-9f0a-994d78990287/volumes" Nov 01 21:41:22.986429 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:22.986394 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:41:23.215194 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:23.215160 5633 generic.go:356] "Generic (PLEG): container finished" podID="974fade6-413b-40c0-b6ad-0d206ccbb013" containerID="fc4510dbdf17636c2af3263dd01bc102679abd0f3922e0f6d6c0dbf5534c6fff" exitCode=0 Nov 01 21:41:23.215408 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:23.215226 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" event={"ID":"974fade6-413b-40c0-b6ad-0d206ccbb013","Type":"ContainerDied","Data":"fc4510dbdf17636c2af3263dd01bc102679abd0f3922e0f6d6c0dbf5534c6fff"} Nov 01 21:41:24.367415 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.367387 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:24.466345 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.466305 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-xks-ca\") pod \"974fade6-413b-40c0-b6ad-0d206ccbb013\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " Nov 01 21:41:24.466544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.466440 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974fade6-413b-40c0-b6ad-0d206ccbb013-kube-api-access-dcsdl\" (UniqueName: \"kubernetes.io/projected/974fade6-413b-40c0-b6ad-0d206ccbb013-kube-api-access-dcsdl\") pod \"974fade6-413b-40c0-b6ad-0d206ccbb013\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " Nov 01 21:41:24.466544 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.466495 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-openshift-ca\") pod \"974fade6-413b-40c0-b6ad-0d206ccbb013\" (UID: \"974fade6-413b-40c0-b6ad-0d206ccbb013\") " Nov 01 21:41:24.466873 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.466847 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-openshift-ca" pod "974fade6-413b-40c0-b6ad-0d206ccbb013" (UID: "974fade6-413b-40c0-b6ad-0d206ccbb013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:41:24.468562 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.468539 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-xks-ca" pod "974fade6-413b-40c0-b6ad-0d206ccbb013" (UID: "974fade6-413b-40c0-b6ad-0d206ccbb013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:41:24.468730 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.468709 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974fade6-413b-40c0-b6ad-0d206ccbb013-kube-api-access-dcsdl" pod "974fade6-413b-40c0-b6ad-0d206ccbb013" (UID: "974fade6-413b-40c0-b6ad-0d206ccbb013"). InnerVolumeSpecName "kube-api-access-dcsdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:41:24.568030 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.567996 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcsdl\" (UniqueName: \"kubernetes.io/projected/974fade6-413b-40c0-b6ad-0d206ccbb013-kube-api-access-dcsdl\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:24.568030 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.568025 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:24.568030 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:24.568037 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974fade6-413b-40c0-b6ad-0d206ccbb013-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:41:25.228990 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:25.228961 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" Nov 01 21:41:25.229178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:25.229004 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf" event={"ID":"974fade6-413b-40c0-b6ad-0d206ccbb013","Type":"ContainerDied","Data":"2747b103c12c817c174c63723fa1bd6920a22c0e15089bf34a20af22f4515e45"} Nov 01 21:41:25.229178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:25.229053 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2747b103c12c817c174c63723fa1bd6920a22c0e15089bf34a20af22f4515e45" Nov 01 21:41:31.043709 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:31.043676 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg"] Nov 01 21:41:31.093111 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:31.093080 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vjckg"] Nov 01 21:41:32.466934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:32.466899 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf" path="/var/lib/kubelet/pods/ee401f2b-148d-435a-9d6e-5e3c6cf0fcbf/volumes" Nov 01 21:41:35.570312 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:35.570280 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:41:42.387406 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:42.387366 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:42.501706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:42.501661 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:46.156274 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:41:46.156233 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:41:51.252483 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:51.252444 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:41:53.319397 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:41:53.319360 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:42:16.156621 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:42:16.156581 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:42:19.058848 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:19.058815 5633 scope.go:122] "RemoveContainer" containerID="3442604bddd06b367e67b566078e41fcbb59f092bd1e02572731fad926786e34" Nov 01 21:42:19.068590 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:19.068564 5633 scope.go:122] "RemoveContainer" containerID="108b08ba38def795a157c785e9101077b14e56ebf3bbe7396252cdd7ced5f179" Nov 01 21:42:19.077658 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:19.077638 5633 scope.go:122] "RemoveContainer" containerID="3cc4ea5f02d3d56eb34740979567568ce7af38d6d4df6bd05346d921a2d51387" Nov 01 21:42:44.750771 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:44.750682 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:42:44.897195 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:44.897162 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:42:46.155910 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:42:46.155870 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:42:51.597696 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:51.597650 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:42:55.045060 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:55.045018 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw"] Nov 01 21:42:55.094194 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:55.094154 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-spdfw"] Nov 01 21:42:56.465970 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:42:56.465936 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01e4729-eb41-42f9-9b16-e2a10c6cd0ee" path="/var/lib/kubelet/pods/e01e4729-eb41-42f9-9b16-e2a10c6cd0ee/volumes" Nov 01 21:43:05.046200 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:05.046135 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n"] Nov 01 21:43:05.093873 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:05.093830 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-w9x4n"] Nov 01 21:43:06.467247 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:06.467197 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6820afce-a3c9-4a24-930b-04d83b43a8c2" path="/var/lib/kubelet/pods/6820afce-a3c9-4a24-930b-04d83b43a8c2/volumes" Nov 01 21:43:08.058499 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:08.058455 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:43:10.118845 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.118805 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts"] Nov 01 21:43:10.119351 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.119320 5633 cpu_manager.go:414] "containerMap: removing container" podUID="974fade6-413b-40c0-b6ad-0d206ccbb013" containerName="revoke-keys" Nov 01 21:43:10.119421 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.119354 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974fade6-413b-40c0-b6ad-0d206ccbb013" containerName="revoke-keys" Nov 01 21:43:10.119461 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.119438 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="974fade6-413b-40c0-b6ad-0d206ccbb013" containerName="revoke-keys" Nov 01 21:43:10.122830 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.122812 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.179262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.179227 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts"] Nov 01 21:43:10.224179 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.224144 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:43:10.224179 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.224152 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:43:10.224451 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.224163 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:43:10.224451 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.224224 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:43:10.278406 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.278363 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.278625 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.278415 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.278625 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.278524 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhsxp\" (UniqueName: \"kubernetes.io/projected/54fb8633-082d-4e75-9a34-7313bc30f343-kube-api-access-xhsxp\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.379514 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.379429 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.379514 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.379476 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.379514 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.379498 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhsxp\" (UniqueName: \"kubernetes.io/projected/54fb8633-082d-4e75-9a34-7313bc30f343-kube-api-access-xhsxp\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.380161 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.380133 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.382155 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.382122 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.435846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.435811 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhsxp\" (UniqueName: \"kubernetes.io/projected/54fb8633-082d-4e75-9a34-7313bc30f343-kube-api-access-xhsxp\") pod \"maas-api-revoke-sub-d576e68b-ggpts\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:10.735351 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:10.735292 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:11.108379 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:11.108344 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts"] Nov 01 21:43:11.109778 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:43:11.109749 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fb8633_082d_4e75_9a34_7313bc30f343.slice/crio-77629ad904b573b1d631e2888da6a067063c4103bfe58fa35bbee0862f1540a3 WatchSource:0}: Error finding container 77629ad904b573b1d631e2888da6a067063c4103bfe58fa35bbee0862f1540a3: Status 404 returned error can't find the container with id 77629ad904b573b1d631e2888da6a067063c4103bfe58fa35bbee0862f1540a3 Nov 01 21:43:11.901629 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:11.901589 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" event={"ID":"54fb8633-082d-4e75-9a34-7313bc30f343","Type":"ContainerStarted","Data":"806d2828b7157584755e18da2d549155feefcf2837e6106f6bb1f30b7cb5c440"} Nov 01 21:43:11.902097 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:11.901642 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" event={"ID":"54fb8633-082d-4e75-9a34-7313bc30f343","Type":"ContainerStarted","Data":"77629ad904b573b1d631e2888da6a067063c4103bfe58fa35bbee0862f1540a3"} Nov 01 21:43:12.017044 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:12.016981 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" podStartSLOduration=2.016967649 podStartE2EDuration="2.016967649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:12.015131526 +0000 UTC m=+5999.689390782" watchObservedRunningTime="2026-11-01 21:43:12.016967649 +0000 UTC m=+5999.691226903" Nov 01 21:43:12.909862 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:12.909824 5633 generic.go:356] "Generic (PLEG): container finished" podID="54fb8633-082d-4e75-9a34-7313bc30f343" containerID="806d2828b7157584755e18da2d549155feefcf2837e6106f6bb1f30b7cb5c440" exitCode=0 Nov 01 21:43:12.910282 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:12.909906 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" event={"ID":"54fb8633-082d-4e75-9a34-7313bc30f343","Type":"ContainerDied","Data":"806d2828b7157584755e18da2d549155feefcf2837e6106f6bb1f30b7cb5c440"} Nov 01 21:43:14.058651 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.058627 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:14.219800 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.219772 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-openshift-ca\") pod \"54fb8633-082d-4e75-9a34-7313bc30f343\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " Nov 01 21:43:14.219984 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.219843 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fb8633-082d-4e75-9a34-7313bc30f343-kube-api-access-xhsxp\" (UniqueName: \"kubernetes.io/projected/54fb8633-082d-4e75-9a34-7313bc30f343-kube-api-access-xhsxp\") pod \"54fb8633-082d-4e75-9a34-7313bc30f343\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " Nov 01 21:43:14.219984 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.219973 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-xks-ca\") pod \"54fb8633-082d-4e75-9a34-7313bc30f343\" (UID: \"54fb8633-082d-4e75-9a34-7313bc30f343\") " Nov 01 21:43:14.220215 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.220195 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-openshift-ca" pod "54fb8633-082d-4e75-9a34-7313bc30f343" (UID: "54fb8633-082d-4e75-9a34-7313bc30f343"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:43:14.220443 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.220424 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:43:14.222140 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.222113 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fb8633-082d-4e75-9a34-7313bc30f343-kube-api-access-xhsxp" pod "54fb8633-082d-4e75-9a34-7313bc30f343" (UID: "54fb8633-082d-4e75-9a34-7313bc30f343"). InnerVolumeSpecName "kube-api-access-xhsxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:43:14.222259 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.222173 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-xks-ca" pod "54fb8633-082d-4e75-9a34-7313bc30f343" (UID: "54fb8633-082d-4e75-9a34-7313bc30f343"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:43:14.321167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.321129 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54fb8633-082d-4e75-9a34-7313bc30f343-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:43:14.321167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.321160 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhsxp\" (UniqueName: \"kubernetes.io/projected/54fb8633-082d-4e75-9a34-7313bc30f343-kube-api-access-xhsxp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:43:14.922900 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.922814 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" event={"ID":"54fb8633-082d-4e75-9a34-7313bc30f343","Type":"ContainerDied","Data":"77629ad904b573b1d631e2888da6a067063c4103bfe58fa35bbee0862f1540a3"} Nov 01 21:43:14.922900 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.922875 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77629ad904b573b1d631e2888da6a067063c4103bfe58fa35bbee0862f1540a3" Nov 01 21:43:14.922900 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:14.922827 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts" Nov 01 21:43:16.156140 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:43:16.156102 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:43:19.113406 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:19.113366 5633 scope.go:122] "RemoveContainer" containerID="31ee97080156249b901b326178a34e77575f5f8a4757b62d235fc812cb32d791" Nov 01 21:43:19.122651 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:19.122625 5633 scope.go:122] "RemoveContainer" containerID="1d3861855111ae2e3278c272070630dc3ab2a349056e72935e4026a98cb049f4" Nov 01 21:43:19.135046 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:19.135023 5633 scope.go:122] "RemoveContainer" containerID="eb750ed844042c7af5c66e3ca0923191176d09c36d68c9785ba6310cf1874066" Nov 01 21:43:22.899793 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:22.899754 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:43:31.769447 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.769406 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:43:31.920918 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.920874 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv"] Nov 01 21:43:31.921486 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.921469 5633 cpu_manager.go:414] "containerMap: removing container" podUID="54fb8633-082d-4e75-9a34-7313bc30f343" containerName="revoke-keys" Nov 01 21:43:31.921560 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.921488 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fb8633-082d-4e75-9a34-7313bc30f343" containerName="revoke-keys" Nov 01 21:43:31.921601 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.921572 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fb8633-082d-4e75-9a34-7313bc30f343" containerName="revoke-keys" Nov 01 21:43:31.925559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.925530 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:31.980228 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.980186 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv"] Nov 01 21:43:31.986918 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.986881 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:31.987116 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.986933 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:31.987197 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:31.987124 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkfq\" (UniqueName: \"kubernetes.io/projected/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-kube-api-access-8jkfq\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.025840 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.025736 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:43:32.025840 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.025773 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:43:32.026992 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.026962 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:43:32.026992 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.026991 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:43:32.088174 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.088137 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkfq\" (UniqueName: \"kubernetes.io/projected/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-kube-api-access-8jkfq\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.088386 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.088246 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.088386 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.088269 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.089072 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.089052 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.090973 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.090949 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.145128 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.145094 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkfq\" (UniqueName: \"kubernetes.io/projected/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-kube-api-access-8jkfq\") pod \"maas-api-revoke-sub-d2bc93bc-7b9rv\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.239249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.239217 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:32.613281 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:32.613252 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv"] Nov 01 21:43:32.615003 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:43:32.614969 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac5f194_ebb8_4f8a_81cf_f623baa7b05d.slice/crio-7708275a582fbe20fe0434d288c3981572558c07be3f44a300faacd3293b15c6 WatchSource:0}: Error finding container 7708275a582fbe20fe0434d288c3981572558c07be3f44a300faacd3293b15c6: Status 404 returned error can't find the container with id 7708275a582fbe20fe0434d288c3981572558c07be3f44a300faacd3293b15c6 Nov 01 21:43:33.069177 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:33.069139 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" event={"ID":"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d","Type":"ContainerStarted","Data":"1f7ba44fcb49443430d60cd29e7951656eb962ef69ffc35c03ee6712a9fb7220"} Nov 01 21:43:33.069632 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:33.069202 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" event={"ID":"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d","Type":"ContainerStarted","Data":"7708275a582fbe20fe0434d288c3981572558c07be3f44a300faacd3293b15c6"} Nov 01 21:43:33.180976 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:33.180868 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" podStartSLOduration=2.180851389 podStartE2EDuration="2.180851389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:33.179785412 +0000 UTC m=+6020.854044668" watchObservedRunningTime="2026-11-01 21:43:33.180851389 +0000 UTC m=+6020.855110643" Nov 01 21:43:34.076606 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:34.076568 5633 generic.go:356] "Generic (PLEG): container finished" podID="5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" containerID="1f7ba44fcb49443430d60cd29e7951656eb962ef69ffc35c03ee6712a9fb7220" exitCode=0 Nov 01 21:43:34.077012 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:34.076650 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" event={"ID":"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d","Type":"ContainerDied","Data":"1f7ba44fcb49443430d60cd29e7951656eb962ef69ffc35c03ee6712a9fb7220"} Nov 01 21:43:35.232621 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.232595 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:35.320313 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.320278 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-openshift-ca\") pod \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " Nov 01 21:43:35.320534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.320365 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-kube-api-access-8jkfq\" (UniqueName: \"kubernetes.io/projected/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-kube-api-access-8jkfq\") pod \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " Nov 01 21:43:35.320534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.320465 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-xks-ca\") pod \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\" (UID: \"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d\") " Nov 01 21:43:35.320763 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.320735 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-openshift-ca" pod "5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" (UID: "5ac5f194-ebb8-4f8a-81cf-f623baa7b05d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:43:35.322665 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.322635 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-kube-api-access-8jkfq" pod "5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" (UID: "5ac5f194-ebb8-4f8a-81cf-f623baa7b05d"). InnerVolumeSpecName "kube-api-access-8jkfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:43:35.322805 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.322786 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-xks-ca" pod "5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" (UID: "5ac5f194-ebb8-4f8a-81cf-f623baa7b05d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:43:35.422316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.422232 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:43:35.422316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.422262 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkfq\" (UniqueName: \"kubernetes.io/projected/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-kube-api-access-8jkfq\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:43:35.422316 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:35.422272 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:43:36.089702 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:36.089673 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" Nov 01 21:43:36.089896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:36.089712 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv" event={"ID":"5ac5f194-ebb8-4f8a-81cf-f623baa7b05d","Type":"ContainerDied","Data":"7708275a582fbe20fe0434d288c3981572558c07be3f44a300faacd3293b15c6"} Nov 01 21:43:36.089896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:36.089757 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7708275a582fbe20fe0434d288c3981572558c07be3f44a300faacd3293b15c6" Nov 01 21:43:40.282451 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:40.282408 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:43:46.162853 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:43:46.162806 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:43:51.107777 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:51.107677 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:51.299002 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:51.298959 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:51.932277 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:51.932243 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:43:56.074469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:43:56.074429 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:44:16.160947 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:44:16.160898 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:44:19.175964 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:44:19.174324 5633 scope.go:122] "RemoveContainer" containerID="9d836f833d58ae7eae82b325474944b51c02ac1bf75fc28f5589fbfd3cda9c07" Nov 01 21:44:19.204007 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:44:19.203980 5633 scope.go:122] "RemoveContainer" containerID="ec6286c7a4dd4ba3ef4c40a0342d992c12ca1d489ace2183731e2a7a2944d45d" Nov 01 21:44:46.157552 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:44:46.157507 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:44:52.304784 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:44:52.304746 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:58.046195 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:44:58.046154 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw"] Nov 01 21:44:58.095382 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:44:58.095317 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91c95d31-z4mnw"] Nov 01 21:44:58.467404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:44:58.467369 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37df3fa9-0c07-4091-bdcb-77cb13d05f2b" path="/var/lib/kubelet/pods/37df3fa9-0c07-4091-bdcb-77cb13d05f2b/volumes" Nov 01 21:45:00.160270 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.160218 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq"] Nov 01 21:45:00.160844 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.160746 5633 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" containerName="revoke-keys" Nov 01 21:45:00.160844 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.160760 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" containerName="revoke-keys" Nov 01 21:45:00.160844 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.160835 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" containerName="revoke-keys" Nov 01 21:45:00.164424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.164392 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.216389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.216319 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5fm2\" (UniqueName: \"kubernetes.io/projected/17f1c55e-1137-4222-93f2-63c92e871ed1-kube-api-access-s5fm2\") pod \"maas-api-key-cleanup-29892825-fpxqq\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.216605 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.216484 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1c55e-1137-4222-93f2-63c92e871ed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892825-fpxqq\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.220107 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.220072 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq"] Nov 01 21:45:00.223251 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.223226 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:45:00.223443 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.223297 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:45:00.264528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.264497 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:45:00.317660 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.317619 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1c55e-1137-4222-93f2-63c92e871ed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892825-fpxqq\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.317868 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.317807 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5fm2\" (UniqueName: \"kubernetes.io/projected/17f1c55e-1137-4222-93f2-63c92e871ed1-kube-api-access-s5fm2\") pod \"maas-api-key-cleanup-29892825-fpxqq\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.318489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.318460 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1c55e-1137-4222-93f2-63c92e871ed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892825-fpxqq\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.374376 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.374320 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5fm2\" (UniqueName: \"kubernetes.io/projected/17f1c55e-1137-4222-93f2-63c92e871ed1-kube-api-access-s5fm2\") pod \"maas-api-key-cleanup-29892825-fpxqq\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.478224 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.478189 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:00.859825 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:00.859799 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq"] Nov 01 21:45:00.861094 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:45:00.861063 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f1c55e_1137_4222_93f2_63c92e871ed1.slice/crio-8b945279dc8379218d16da6855c0efb0361747cc20a65e38da52812257975f8d WatchSource:0}: Error finding container 8b945279dc8379218d16da6855c0efb0361747cc20a65e38da52812257975f8d: Status 404 returned error can't find the container with id 8b945279dc8379218d16da6855c0efb0361747cc20a65e38da52812257975f8d Nov 01 21:45:01.703980 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:01.703939 5633 generic.go:356] "Generic (PLEG): container finished" podID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerID="2ca70fd539175ee5f243b9fcbe2ec3bc7c562310880754dc3cd130aa6c839047" exitCode=60 Nov 01 21:45:01.704484 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:01.703993 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" event={"ID":"17f1c55e-1137-4222-93f2-63c92e871ed1","Type":"ContainerDied","Data":"2ca70fd539175ee5f243b9fcbe2ec3bc7c562310880754dc3cd130aa6c839047"} Nov 01 21:45:01.704484 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:01.704068 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" event={"ID":"17f1c55e-1137-4222-93f2-63c92e871ed1","Type":"ContainerStarted","Data":"8b945279dc8379218d16da6855c0efb0361747cc20a65e38da52812257975f8d"} Nov 01 21:45:01.704484 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:01.704422 5633 scope.go:122] "RemoveContainer" containerID="2ca70fd539175ee5f243b9fcbe2ec3bc7c562310880754dc3cd130aa6c839047" Nov 01 21:45:02.712413 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:02.712371 5633 generic.go:356] "Generic (PLEG): container finished" podID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerID="35bd3c6b599dc8cef2083d906e24677655e386d23df21fb00e25d5b1c3e0fe47" exitCode=60 Nov 01 21:45:02.712413 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:02.712427 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" event={"ID":"17f1c55e-1137-4222-93f2-63c92e871ed1","Type":"ContainerDied","Data":"35bd3c6b599dc8cef2083d906e24677655e386d23df21fb00e25d5b1c3e0fe47"} Nov 01 21:45:02.713012 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:02.712469 5633 scope.go:122] "RemoveContainer" containerID="2ca70fd539175ee5f243b9fcbe2ec3bc7c562310880754dc3cd130aa6c839047" Nov 01 21:45:02.713012 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:02.712751 5633 scope.go:122] "RemoveContainer" containerID="35bd3c6b599dc8cef2083d906e24677655e386d23df21fb00e25d5b1c3e0fe47" Nov 01 21:45:02.713130 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:45:02.713068 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892825-fpxqq_odh-ai-gateway-infra(17f1c55e-1137-4222-93f2-63c92e871ed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" Nov 01 21:45:03.720397 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:03.720367 5633 scope.go:122] "RemoveContainer" containerID="35bd3c6b599dc8cef2083d906e24677655e386d23df21fb00e25d5b1c3e0fe47" Nov 01 21:45:03.720795 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:45:03.720706 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892825-fpxqq_odh-ai-gateway-infra(17f1c55e-1137-4222-93f2-63c92e871ed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" Nov 01 21:45:14.459987 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:14.459942 5633 scope.go:122] "RemoveContainer" containerID="35bd3c6b599dc8cef2083d906e24677655e386d23df21fb00e25d5b1c3e0fe47" Nov 01 21:45:15.792521 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:15.792487 5633 generic.go:356] "Generic (PLEG): container finished" podID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerID="0d61b5ff22576a6b8d8217c667f445b1edfea3175891162f0a744479ea6ef9d8" exitCode=60 Nov 01 21:45:15.793016 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:15.792534 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" event={"ID":"17f1c55e-1137-4222-93f2-63c92e871ed1","Type":"ContainerDied","Data":"0d61b5ff22576a6b8d8217c667f445b1edfea3175891162f0a744479ea6ef9d8"} Nov 01 21:45:15.793016 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:15.792583 5633 scope.go:122] "RemoveContainer" containerID="35bd3c6b599dc8cef2083d906e24677655e386d23df21fb00e25d5b1c3e0fe47" Nov 01 21:45:15.793204 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:15.793038 5633 scope.go:122] "RemoveContainer" containerID="0d61b5ff22576a6b8d8217c667f445b1edfea3175891162f0a744479ea6ef9d8" Nov 01 21:45:15.793354 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:45:15.793299 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892825-fpxqq_odh-ai-gateway-infra(17f1c55e-1137-4222-93f2-63c92e871ed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" Nov 01 21:45:16.156251 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:45:16.156165 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:45:16.916900 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:16.916862 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq"] Nov 01 21:45:17.077430 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.077398 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:17.210404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.210008 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f1c55e-1137-4222-93f2-63c92e871ed1-kube-api-access-s5fm2\" (UniqueName: \"kubernetes.io/projected/17f1c55e-1137-4222-93f2-63c92e871ed1-kube-api-access-s5fm2\") pod \"17f1c55e-1137-4222-93f2-63c92e871ed1\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " Nov 01 21:45:17.210404 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.210082 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f1c55e-1137-4222-93f2-63c92e871ed1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1c55e-1137-4222-93f2-63c92e871ed1-maas-api-service-ca\") pod \"17f1c55e-1137-4222-93f2-63c92e871ed1\" (UID: \"17f1c55e-1137-4222-93f2-63c92e871ed1\") " Nov 01 21:45:17.214353 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.210987 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f1c55e-1137-4222-93f2-63c92e871ed1-maas-api-service-ca" pod "17f1c55e-1137-4222-93f2-63c92e871ed1" (UID: "17f1c55e-1137-4222-93f2-63c92e871ed1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:17.227355 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.225983 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f1c55e-1137-4222-93f2-63c92e871ed1-kube-api-access-s5fm2" pod "17f1c55e-1137-4222-93f2-63c92e871ed1" (UID: "17f1c55e-1137-4222-93f2-63c92e871ed1"). InnerVolumeSpecName "kube-api-access-s5fm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:17.312026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.311990 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5fm2\" (UniqueName: \"kubernetes.io/projected/17f1c55e-1137-4222-93f2-63c92e871ed1-kube-api-access-s5fm2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:45:17.312026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.312026 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f1c55e-1137-4222-93f2-63c92e871ed1-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:45:17.811705 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.811675 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" Nov 01 21:45:17.811705 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.811682 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq" event={"ID":"17f1c55e-1137-4222-93f2-63c92e871ed1","Type":"ContainerDied","Data":"8b945279dc8379218d16da6855c0efb0361747cc20a65e38da52812257975f8d"} Nov 01 21:45:17.811962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.811760 5633 scope.go:122] "RemoveContainer" containerID="0d61b5ff22576a6b8d8217c667f445b1edfea3175891162f0a744479ea6ef9d8" Nov 01 21:45:17.998214 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:17.998179 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq"] Nov 01 21:45:18.004988 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:18.004957 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892825-fpxqq"] Nov 01 21:45:18.468729 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:18.468695 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" path="/var/lib/kubelet/pods/17f1c55e-1137-4222-93f2-63c92e871ed1/volumes" Nov 01 21:45:19.234216 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:19.234170 5633 scope.go:122] "RemoveContainer" containerID="90591b3430e6958f634751f69b1a3a187f17e0c398e38332b333fba5bb04331c" Nov 01 21:45:46.158087 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:45:46.157985 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:45:52.728804 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:45:52.728761 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:16.159418 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:46:16.159374 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:46:19.264136 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:19.264095 5633 scope.go:122] "RemoveContainer" containerID="34fc5951ef02b4d2568ceb940a002fbb35b8d1f83be7bfd6c9b848c213adcc10" Nov 01 21:46:25.046482 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:25.046448 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf"] Nov 01 21:46:25.095785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:25.095750 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-brnsf"] Nov 01 21:46:26.465798 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:26.465760 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974fade6-413b-40c0-b6ad-0d206ccbb013" path="/var/lib/kubelet/pods/974fade6-413b-40c0-b6ad-0d206ccbb013/volumes" Nov 01 21:46:46.156462 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:46:46.156424 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:46:52.968169 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:52.968078 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:46:57.704534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.704497 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn"] Nov 01 21:46:57.705191 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705170 5633 cpu_manager.go:414] "containerMap: removing container" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.705287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705194 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.705287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705259 5633 cpu_manager.go:414] "containerMap: removing container" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.705287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705267 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.705287 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705280 5633 cpu_manager.go:414] "containerMap: removing container" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.705546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705288 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.705546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705419 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.705546 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.705443 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:46:57.708904 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.708884 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.760754 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.760720 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:46:57.760754 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.760720 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:46:57.760987 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.760770 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:46:57.810134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.810097 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:46:57.813480 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.813453 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn"] Nov 01 21:46:57.845880 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.845844 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.846065 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.845896 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cp7m\" (UniqueName: \"kubernetes.io/projected/cae1e876-70c6-482f-8d91-a84dbc36ebbd-kube-api-access-7cp7m\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.846065 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.846024 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.946995 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.946957 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.947184 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.947008 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cp7m\" (UniqueName: \"kubernetes.io/projected/cae1e876-70c6-482f-8d91-a84dbc36ebbd-kube-api-access-7cp7m\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.947184 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.947059 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.947833 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.947812 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:57.949778 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:57.949741 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:58.003262 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:58.003232 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cp7m\" (UniqueName: \"kubernetes.io/projected/cae1e876-70c6-482f-8d91-a84dbc36ebbd-kube-api-access-7cp7m\") pod \"maas-api-revoke-sub-782ed212-kfztn\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:58.022113 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:58.022081 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:46:58.394533 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:58.394504 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn"] Nov 01 21:46:58.396535 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:46:58.396503 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae1e876_70c6_482f_8d91_a84dbc36ebbd.slice/crio-97dc89031c17c4b59d66ce1d76b5589bc1b390dd17f55571ff2c5884f69cce9c WatchSource:0}: Error finding container 97dc89031c17c4b59d66ce1d76b5589bc1b390dd17f55571ff2c5884f69cce9c: Status 404 returned error can't find the container with id 97dc89031c17c4b59d66ce1d76b5589bc1b390dd17f55571ff2c5884f69cce9c Nov 01 21:46:58.436542 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:58.436513 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" event={"ID":"cae1e876-70c6-482f-8d91-a84dbc36ebbd","Type":"ContainerStarted","Data":"97dc89031c17c4b59d66ce1d76b5589bc1b390dd17f55571ff2c5884f69cce9c"} Nov 01 21:46:59.444011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:59.443972 5633 generic.go:356] "Generic (PLEG): container finished" podID="cae1e876-70c6-482f-8d91-a84dbc36ebbd" containerID="ab27c46a418782096879cd20f13466692d45881287ac3e06e1717e9254f5f83a" exitCode=0 Nov 01 21:46:59.444495 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:46:59.444048 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" event={"ID":"cae1e876-70c6-482f-8d91-a84dbc36ebbd","Type":"ContainerDied","Data":"ab27c46a418782096879cd20f13466692d45881287ac3e06e1717e9254f5f83a"} Nov 01 21:47:00.599959 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.599931 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:47:00.678705 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.678671 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-xks-ca\") pod \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " Nov 01 21:47:00.678924 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.678777 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae1e876-70c6-482f-8d91-a84dbc36ebbd-kube-api-access-7cp7m\" (UniqueName: \"kubernetes.io/projected/cae1e876-70c6-482f-8d91-a84dbc36ebbd-kube-api-access-7cp7m\") pod \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " Nov 01 21:47:00.678924 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.678847 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-openshift-ca\") pod \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\" (UID: \"cae1e876-70c6-482f-8d91-a84dbc36ebbd\") " Nov 01 21:47:00.679232 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.679208 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-openshift-ca" pod "cae1e876-70c6-482f-8d91-a84dbc36ebbd" (UID: "cae1e876-70c6-482f-8d91-a84dbc36ebbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:47:00.680922 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.680898 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-xks-ca" pod "cae1e876-70c6-482f-8d91-a84dbc36ebbd" (UID: "cae1e876-70c6-482f-8d91-a84dbc36ebbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:47:00.681080 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.681057 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae1e876-70c6-482f-8d91-a84dbc36ebbd-kube-api-access-7cp7m" pod "cae1e876-70c6-482f-8d91-a84dbc36ebbd" (UID: "cae1e876-70c6-482f-8d91-a84dbc36ebbd"). InnerVolumeSpecName "kube-api-access-7cp7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:47:00.780362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.780295 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cp7m\" (UniqueName: \"kubernetes.io/projected/cae1e876-70c6-482f-8d91-a84dbc36ebbd-kube-api-access-7cp7m\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:47:00.780362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.780342 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:47:00.780362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:00.780353 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae1e876-70c6-482f-8d91-a84dbc36ebbd-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:47:01.457773 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:01.457738 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" Nov 01 21:47:01.457937 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:01.457774 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn" event={"ID":"cae1e876-70c6-482f-8d91-a84dbc36ebbd","Type":"ContainerDied","Data":"97dc89031c17c4b59d66ce1d76b5589bc1b390dd17f55571ff2c5884f69cce9c"} Nov 01 21:47:01.457937 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:01.457821 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97dc89031c17c4b59d66ce1d76b5589bc1b390dd17f55571ff2c5884f69cce9c" Nov 01 21:47:08.550381 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:08.550341 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:12.388672 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:12.388633 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:47:14.169147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:14.169102 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:16.156804 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:47:16.156762 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:47:19.286367 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:19.286316 5633 scope.go:122] "RemoveContainer" containerID="116b66a27a95c4ecb4cee4b80ad7a49e427907657f6c6e20cd357b4bb9208748" Nov 01 21:47:25.823498 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:25.823449 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:25.990911 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:25.990873 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:47:46.155935 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:47:46.155898 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:47:53.305344 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:47:53.305302 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:15.054240 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:15.054196 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts"] Nov 01 21:48:15.100622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:15.100580 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ggpts"] Nov 01 21:48:16.159435 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:48:16.159394 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:48:16.466294 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:16.466259 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fb8633-082d-4e75-9a34-7313bc30f343" path="/var/lib/kubelet/pods/54fb8633-082d-4e75-9a34-7313bc30f343/volumes" Nov 01 21:48:19.309555 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:19.309521 5633 scope.go:122] "RemoveContainer" containerID="fc4510dbdf17636c2af3263dd01bc102679abd0f3922e0f6d6c0dbf5534c6fff" Nov 01 21:48:36.044373 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:36.044284 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv"] Nov 01 21:48:36.092903 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:36.092867 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7b9rv"] Nov 01 21:48:36.466152 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:36.466120 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac5f194-ebb8-4f8a-81cf-f623baa7b05d" path="/var/lib/kubelet/pods/5ac5f194-ebb8-4f8a-81cf-f623baa7b05d/volumes" Nov 01 21:48:38.972568 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:38.972532 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:39.156069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:39.156028 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:39.966618 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:39.966577 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:48:46.160176 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:48:46.160135 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:48:53.703415 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:48:53.703377 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:06.178653 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:49:06.178617 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:12.012894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:49:12.012849 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:49:16.157743 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:49:16.157705 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:49:19.335146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:49:19.335114 5633 scope.go:122] "RemoveContainer" containerID="806d2828b7157584755e18da2d549155feefcf2837e6106f6bb1f30b7cb5c440" Nov 01 21:49:46.156511 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:49:46.156470 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:49:53.992805 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:49:53.992719 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:16.158462 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:50:16.158424 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:50:19.359869 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:50:19.359832 5633 scope.go:122] "RemoveContainer" containerID="1f7ba44fcb49443430d60cd29e7951656eb962ef69ffc35c03ee6712a9fb7220" Nov 01 21:50:36.602974 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:50:36.602935 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:36.739395 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:50:36.739358 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:46.157418 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:50:46.157379 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:50:53.277392 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:50:53.277352 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:54.330195 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:50:54.330161 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:50:59.025304 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:50:59.025267 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:51:16.156581 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:51:16.156529 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:51:46.157612 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:51:46.157527 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:51:54.671069 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:51:54.671029 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:01.043958 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:01.043920 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn"] Nov 01 21:52:01.093146 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:01.093108 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ed212-kfztn"] Nov 01 21:52:02.465691 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:02.465654 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae1e876-70c6-482f-8d91-a84dbc36ebbd" path="/var/lib/kubelet/pods/cae1e876-70c6-482f-8d91-a84dbc36ebbd/volumes" Nov 01 21:52:16.155636 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:52:16.155588 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:52:23.434167 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:23.434127 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:23.564622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:23.564586 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:34.398697 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:34.398661 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:40.402322 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:40.402275 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:45.965643 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:45.965604 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:46.157183 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:52:46.157144 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:52:51.379293 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:51.379203 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t"] Nov 01 21:52:51.379814 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:51.379525 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" containerID="cri-o://48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d" gracePeriod=30 Nov 01 21:52:51.429041 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:51.429005 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c"] Nov 01 21:52:51.429259 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:51.429238 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" containerID="cri-o://c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd" gracePeriod=30 Nov 01 21:52:55.013872 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:55.013833 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:52:56.678378 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.678293 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:52:56.711635 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.711610 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:52:56.826706 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826676 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b8eae30-2493-4455-8868-4230de1d9526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8eae30-2493-4455-8868-4230de1d9526-istio-podinfo\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.826913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826724 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-socket\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.826913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826754 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-data\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.826913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826792 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-kube-api-access-bmbx8\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-kube-api-access-bmbx8\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.826913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826856 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-envoy\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.826913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826881 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-certs\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826931 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-certs\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826968 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istiod-ca-cert\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826996 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-credential-socket\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.826995 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-socket" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827041 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-kube-api-access-78pgb\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-kube-api-access-78pgb\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827082 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-istio-token\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-istio-token\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827126 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-token\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-token\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827084 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-data" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.827193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827184 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-envoy\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.827695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827214 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-socket\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.827695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827247 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-certs" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.827695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827289 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-credential-socket\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.827695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827319 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-podinfo\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.827695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827361 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-certs" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.827695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827378 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-data\") pod \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\" (UID: \"b661e9d8-10bf-490d-9ca9-1093dca1ddc1\") " Nov 01 21:52:56.827695 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827418 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8eae30-2493-4455-8868-4230de1d9526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8eae30-2493-4455-8868-4230de1d9526-istiod-ca-cert\") pod \"3b8eae30-2493-4455-8868-4230de1d9526\" (UID: \"3b8eae30-2493-4455-8868-4230de1d9526\") " Nov 01 21:52:56.828085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827820 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.828085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827791 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istiod-ca-cert" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:52:56.828085 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.827839 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.828264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.828104 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.828264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.828126 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.828264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.828122 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-credential-socket" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.828264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.828160 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8eae30-2493-4455-8868-4230de1d9526-istiod-ca-cert" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:52:56.828548 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.828391 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-credential-socket" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.828621 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.828584 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-socket" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.829701 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.829664 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-data" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.830022 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.829986 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-kube-api-access-bmbx8" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "kube-api-access-bmbx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.830150 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.830050 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b8eae30-2493-4455-8868-4230de1d9526-istio-podinfo" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:52:56.830361 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.830298 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-envoy" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.830929 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.830901 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-podinfo" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:52:56.831056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.831031 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-kube-api-access-78pgb" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "kube-api-access-78pgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.831125 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.831036 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-envoy" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.831125 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.831077 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-token" pod "b661e9d8-10bf-490d-9ca9-1093dca1ddc1" (UID: "b661e9d8-10bf-490d-9ca9-1093dca1ddc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.831980 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.831959 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-istio-token" pod "3b8eae30-2493-4455-8868-4230de1d9526" (UID: "3b8eae30-2493-4455-8868-4230de1d9526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929425 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929456 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929466 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78pgb\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-kube-api-access-78pgb\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929477 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929486 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929493 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929501 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929510 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8eae30-2493-4455-8868-4230de1d9526-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929510 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929518 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929526 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929535 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8eae30-2493-4455-8868-4230de1d9526-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929544 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8eae30-2493-4455-8868-4230de1d9526-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929552 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmbx8\" (UniqueName: \"kubernetes.io/projected/3b8eae30-2493-4455-8868-4230de1d9526-kube-api-access-bmbx8\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.929934 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:56.929560 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b661e9d8-10bf-490d-9ca9-1093dca1ddc1-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:57.464346 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.464300 5633 generic.go:356] "Generic (PLEG): container finished" podID="3b8eae30-2493-4455-8868-4230de1d9526" containerID="48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d" exitCode=0 Nov 01 21:52:57.464558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.464358 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" Nov 01 21:52:57.464558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.464382 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" event={"ID":"3b8eae30-2493-4455-8868-4230de1d9526","Type":"ContainerDied","Data":"48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d"} Nov 01 21:52:57.464558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.464438 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t" event={"ID":"3b8eae30-2493-4455-8868-4230de1d9526","Type":"ContainerDied","Data":"f9c2d7dd91e7cfe0df5134bab3ae312812e84a51fa0adb8eecf13790c7d1eebb"} Nov 01 21:52:57.464558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.464465 5633 scope.go:122] "RemoveContainer" containerID="48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d" Nov 01 21:52:57.467154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.467108 5633 generic.go:356] "Generic (PLEG): container finished" podID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerID="c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd" exitCode=0 Nov 01 21:52:57.467154 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.467150 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" event={"ID":"b661e9d8-10bf-490d-9ca9-1093dca1ddc1","Type":"ContainerDied","Data":"c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd"} Nov 01 21:52:57.467465 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.467163 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" Nov 01 21:52:57.467465 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.467194 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c" event={"ID":"b661e9d8-10bf-490d-9ca9-1093dca1ddc1","Type":"ContainerDied","Data":"465aef1ed890de9ca17eedfbdc23da4a34d3b36d037711ad98ac40e4932316fe"} Nov 01 21:52:57.480011 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.479959 5633 scope.go:122] "RemoveContainer" containerID="48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d" Nov 01 21:52:57.480650 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:52:57.480622 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d\": container with ID starting with 48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d not found: ID does not exist" containerID="48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d" Nov 01 21:52:57.480785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.480657 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d"} err="failed to get container status \"48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d\": rpc error: code = NotFound desc = could not find container \"48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d\": container with ID starting with 48f95822cb2de9b8dc7d6a699f8796eed97fcb0dfdb32e9a689ff6fc1bd7c03d not found: ID does not exist" Nov 01 21:52:57.480785 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.480678 5633 scope.go:122] "RemoveContainer" containerID="c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd" Nov 01 21:52:57.490019 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.489994 5633 scope.go:122] "RemoveContainer" containerID="c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd" Nov 01 21:52:57.490353 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:52:57.490308 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd\": container with ID starting with c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd not found: ID does not exist" containerID="c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd" Nov 01 21:52:57.490430 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.490367 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd"} err="failed to get container status \"c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd\": rpc error: code = NotFound desc = could not find container \"c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd\": container with ID starting with c94324f1946e4d50c4490e1fa172c0e932a95c330e398a6923dff5d2dc7c88fd not found: ID does not exist" Nov 01 21:52:57.626992 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.626956 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c"] Nov 01 21:52:57.675659 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.675622 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9rk4c"] Nov 01 21:52:57.735139 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.735101 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t"] Nov 01 21:52:57.738636 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:57.738608 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fvd8t"] Nov 01 21:52:58.466402 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:58.466367 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8eae30-2493-4455-8868-4230de1d9526" path="/var/lib/kubelet/pods/3b8eae30-2493-4455-8868-4230de1d9526/volumes" Nov 01 21:52:58.466846 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:52:58.466833 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" path="/var/lib/kubelet/pods/b661e9d8-10bf-490d-9ca9-1093dca1ddc1/volumes" Nov 01 21:53:06.494819 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:06.494757 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc"] Nov 01 21:53:06.495388 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:06.495033 5633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" containerID="cri-o://db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e" gracePeriod=30 Nov 01 21:53:11.761356 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.761306 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:53:11.782464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782427 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istiod-ca-cert\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782472 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-podinfo\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782508 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-token\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-token\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782669 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782549 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-kube-api-access-s6rkh\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-kube-api-access-s6rkh\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782680 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-data\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782729 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-socket\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782787 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-certs\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782828 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-envoy\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.782859 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782858 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-credential-socket\") pod \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\" (UID: \"f7ce3ee8-0815-4096-ad3b-f886a79b7032\") " Nov 01 21:53:11.783147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782887 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istiod-ca-cert" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:53:11.783147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.782937 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-data" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:53:11.783147 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.783058 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-socket" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:53:11.783511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.783198 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-certs" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:53:11.783511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.783224 5633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istiod-ca-cert\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.783511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.783256 5633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-data\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.783511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.783272 5633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.783511 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.783276 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-credential-socket" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:53:11.785021 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.784992 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-token" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:53:11.785148 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.785050 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-podinfo" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:53:11.785148 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.785088 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-kube-api-access-s6rkh" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "kube-api-access-s6rkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:53:11.785257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.785233 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-envoy" pod "f7ce3ee8-0815-4096-ad3b-f886a79b7032" (UID: "f7ce3ee8-0815-4096-ad3b-f886a79b7032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:53:11.884074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.884029 5633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-podinfo\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.884074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.884071 5633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-token\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.884283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.884088 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rkh\" (UniqueName: \"kubernetes.io/projected/f7ce3ee8-0815-4096-ad3b-f886a79b7032-kube-api-access-s6rkh\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.884283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.884105 5633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-workload-certs\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.884283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.884120 5633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-istio-envoy\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:11.884283 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:11.884136 5633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ce3ee8-0815-4096-ad3b-f886a79b7032-credential-socket\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:53:12.557467 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.557427 5633 generic.go:356] "Generic (PLEG): container finished" podID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerID="db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e" exitCode=0 Nov 01 21:53:12.557685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.557474 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" Nov 01 21:53:12.557685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.557493 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" event={"ID":"f7ce3ee8-0815-4096-ad3b-f886a79b7032","Type":"ContainerDied","Data":"db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e"} Nov 01 21:53:12.557685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.557540 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc" event={"ID":"f7ce3ee8-0815-4096-ad3b-f886a79b7032","Type":"ContainerDied","Data":"0afc661e503909678aafe3bebc0dc9448e68a55293b3c5da428f800c0135f09d"} Nov 01 21:53:12.557685 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.557564 5633 scope.go:122] "RemoveContainer" containerID="db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e" Nov 01 21:53:12.567445 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.567420 5633 scope.go:122] "RemoveContainer" containerID="db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e" Nov 01 21:53:12.567786 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:53:12.567757 5633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e\": container with ID starting with db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e not found: ID does not exist" containerID="db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e" Nov 01 21:53:12.567908 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.567794 5633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e"} err="failed to get container status \"db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e\": rpc error: code = NotFound desc = could not find container \"db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e\": container with ID starting with db82598f089516f4683273584f707334387e63772d2b30607d65af845acbc54e not found: ID does not exist" Nov 01 21:53:12.719643 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.719604 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc"] Nov 01 21:53:12.768917 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:12.768833 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fjrwc"] Nov 01 21:53:14.466411 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:14.466378 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" path="/var/lib/kubelet/pods/f7ce3ee8-0815-4096-ad3b-f886a79b7032/volumes" Nov 01 21:53:16.156214 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:53:16.156179 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:53:19.399430 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:19.399399 5633 scope.go:122] "RemoveContainer" containerID="ab27c46a418782096879cd20f13466692d45881287ac3e06e1717e9254f5f83a" Nov 01 21:53:46.155542 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:53:46.155502 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:53:55.390694 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:53:55.390648 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:10.872622 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:10.872578 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.005464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:11.005420 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:11.205989 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:11.205893 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:12.370084 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:12.370042 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:12.555615 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:12.555578 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:14.402193 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402148 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc"] Nov 01 21:54:14.402789 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402768 5633 cpu_manager.go:414] "containerMap: removing container" podUID="cae1e876-70c6-482f-8d91-a84dbc36ebbd" containerName="revoke-keys" Nov 01 21:54:14.402881 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402794 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae1e876-70c6-482f-8d91-a84dbc36ebbd" containerName="revoke-keys" Nov 01 21:54:14.402881 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402851 5633 cpu_manager.go:414] "containerMap: removing container" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" Nov 01 21:54:14.402881 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402860 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" Nov 01 21:54:14.402881 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402872 5633 cpu_manager.go:414] "containerMap: removing container" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402882 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402903 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.402910 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.403025 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ce3ee8-0815-4096-ad3b-f886a79b7032" containerName="istio-proxy" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.403036 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b661e9d8-10bf-490d-9ca9-1093dca1ddc1" containerName="istio-proxy" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.403048 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f1c55e-1137-4222-93f2-63c92e871ed1" containerName="cleanup" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.403057 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8eae30-2493-4455-8868-4230de1d9526" containerName="istio-proxy" Nov 01 21:54:14.403104 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.403069 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae1e876-70c6-482f-8d91-a84dbc36ebbd" containerName="revoke-keys" Nov 01 21:54:14.406402 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.406379 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.467243 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.467209 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc"] Nov 01 21:54:14.506853 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.506823 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:54:14.507031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.506998 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:54:14.507031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.507011 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:54:14.507128 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.507103 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.507240 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.507226 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx725\" (UniqueName: \"kubernetes.io/projected/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-kube-api-access-rx725\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.507312 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.507300 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.508042 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.508029 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:54:14.608301 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.608265 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.608528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.608358 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx725\" (UniqueName: \"kubernetes.io/projected/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-kube-api-access-rx725\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.608528 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.608439 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.609075 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.609054 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.610931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.610909 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.665571 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.665469 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx725\" (UniqueName: \"kubernetes.io/projected/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-kube-api-access-rx725\") pod \"maas-api-revoke-sub-ca1f3380-kmbrc\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:14.719833 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:14.719796 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:15.090817 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:15.090790 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc"] Nov 01 21:54:15.092209 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:54:15.092174 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd034cc96_4a6d_48e9_870d_a1dd7a6cd7eb.slice/crio-fe8f389c5a633ba6951b5dea468b3938f578e2f7b2303cbc039a15d837ce87e4 WatchSource:0}: Error finding container fe8f389c5a633ba6951b5dea468b3938f578e2f7b2303cbc039a15d837ce87e4: Status 404 returned error can't find the container with id fe8f389c5a633ba6951b5dea468b3938f578e2f7b2303cbc039a15d837ce87e4 Nov 01 21:54:15.893352 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:15.893295 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" event={"ID":"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb","Type":"ContainerStarted","Data":"e31ef33aa498df070a9b86c1eef0968a6a4ae7cb089adecaedb855a74dbbc287"} Nov 01 21:54:15.893847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:15.893380 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" event={"ID":"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb","Type":"ContainerStarted","Data":"fe8f389c5a633ba6951b5dea468b3938f578e2f7b2303cbc039a15d837ce87e4"} Nov 01 21:54:16.156542 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:54:16.156450 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:54:16.784225 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:16.784163 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" podStartSLOduration=2.78414668 podStartE2EDuration="2.78414668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:16.004771386 +0000 UTC m=+6663.679030642" watchObservedRunningTime="2026-11-01 21:54:16.78414668 +0000 UTC m=+6664.458405934" Nov 01 21:54:16.785340 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:16.785308 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:54:16.899945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:16.899913 5633 generic.go:356] "Generic (PLEG): container finished" podID="d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" containerID="e31ef33aa498df070a9b86c1eef0968a6a4ae7cb089adecaedb855a74dbbc287" exitCode=0 Nov 01 21:54:16.900470 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:16.900009 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" event={"ID":"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb","Type":"ContainerDied","Data":"e31ef33aa498df070a9b86c1eef0968a6a4ae7cb089adecaedb855a74dbbc287"} Nov 01 21:54:18.054054 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.054028 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:18.151196 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.151163 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-xks-ca\") pod \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " Nov 01 21:54:18.151432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.151227 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-openshift-ca\") pod \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " Nov 01 21:54:18.151432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.151368 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-kube-api-access-rx725\" (UniqueName: \"kubernetes.io/projected/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-kube-api-access-rx725\") pod \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\" (UID: \"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb\") " Nov 01 21:54:18.151693 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.151668 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-openshift-ca" pod "d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" (UID: "d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:54:18.151779 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.151761 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:54:18.153566 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.153541 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-xks-ca" pod "d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" (UID: "d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:54:18.153665 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.153651 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-kube-api-access-rx725" pod "d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" (UID: "d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb"). InnerVolumeSpecName "kube-api-access-rx725". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:54:18.253131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.253087 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:54:18.253131 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.253122 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx725\" (UniqueName: \"kubernetes.io/projected/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb-kube-api-access-rx725\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:54:18.915608 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.915533 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" Nov 01 21:54:18.915608 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.915541 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc" event={"ID":"d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb","Type":"ContainerDied","Data":"fe8f389c5a633ba6951b5dea468b3938f578e2f7b2303cbc039a15d837ce87e4"} Nov 01 21:54:18.915608 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:18.915587 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe8f389c5a633ba6951b5dea468b3938f578e2f7b2303cbc039a15d837ce87e4" Nov 01 21:54:24.997828 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:24.997745 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:30.985824 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:30.985785 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:37.186453 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.186415 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg"] Nov 01 21:54:37.186919 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.186904 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" containerName="revoke-keys" Nov 01 21:54:37.186966 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.186921 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" containerName="revoke-keys" Nov 01 21:54:37.187003 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.186998 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" containerName="revoke-keys" Nov 01 21:54:37.190076 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.190061 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.241748 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.241694 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\"" Nov 01 21:54:37.241961 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.241787 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 21:54:37.247196 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247168 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247309 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247232 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247417 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247392 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247476 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247442 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247530 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247512 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247574 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247552 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247571 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddtm\" (UniqueName: \"kubernetes.io/projected/f31af8b8-3cb5-404c-b3ec-ecb67025655b-kube-api-access-jddtm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247628 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247593 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.247721 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.247655 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.348569 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348538 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.348759 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348584 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.348759 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348637 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.348759 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348674 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.348759 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348703 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.348759 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348721 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.348759 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348757 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.349134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348782 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddtm\" (UniqueName: \"kubernetes.io/projected/f31af8b8-3cb5-404c-b3ec-ecb67025655b-kube-api-access-jddtm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.349134 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.348811 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.349249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.349151 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.349249 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.349207 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.349386 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.349261 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.349618 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.349469 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.349618 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.349535 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.351235 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.351212 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.351389 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.351357 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.391317 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.391285 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg"] Nov 01 21:54:37.404734 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.404707 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f31af8b8-3cb5-404c-b3ec-ecb67025655b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.404896 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.404840 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddtm\" (UniqueName: \"kubernetes.io/projected/f31af8b8-3cb5-404c-b3ec-ecb67025655b-kube-api-access-jddtm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fpzcg\" (UID: \"f31af8b8-3cb5-404c-b3ec-ecb67025655b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.502825 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.502790 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:37.929875 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.929846 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg"] Nov 01 21:54:37.931195 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:54:37.931166 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31af8b8_3cb5_404c_b3ec_ecb67025655b.slice/crio-18b70191927a8844727c6fb5ad84ffc2fab926d107720a574b453462782ee661 WatchSource:0}: Error finding container 18b70191927a8844727c6fb5ad84ffc2fab926d107720a574b453462782ee661: Status 404 returned error can't find the container with id 18b70191927a8844727c6fb5ad84ffc2fab926d107720a574b453462782ee661 Nov 01 21:54:37.933265 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.933235 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:54:37.933357 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.933302 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:54:37.933408 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:37.933370 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:54:38.015348 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:38.015290 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" event={"ID":"f31af8b8-3cb5-404c-b3ec-ecb67025655b","Type":"ContainerStarted","Data":"a3dd84aa8c0f600e121638a68ad2c7ad5d68303288d3fe8a3afb83d7aa4ecfeb"} Nov 01 21:54:38.015530 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:38.015415 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" event={"ID":"f31af8b8-3cb5-404c-b3ec-ecb67025655b","Type":"ContainerStarted","Data":"18b70191927a8844727c6fb5ad84ffc2fab926d107720a574b453462782ee661"} Nov 01 21:54:38.132922 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:38.132865 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podStartSLOduration=1.132850122 podStartE2EDuration="1.132850122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:38.130355284 +0000 UTC m=+6685.804614540" watchObservedRunningTime="2026-11-01 21:54:38.132850122 +0000 UTC m=+6685.807109377" Nov 01 21:54:38.503257 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:38.503166 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:39.505253 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:39.505215 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body= Nov 01 21:54:39.505911 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:39.505878 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" Nov 01 21:54:40.503899 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:40.503856 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body= Nov 01 21:54:40.504217 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:40.504186 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" Nov 01 21:54:41.506355 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:41.505772 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body= Nov 01 21:54:41.506355 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:41.505836 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" Nov 01 21:54:42.510507 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:42.510465 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body= Nov 01 21:54:42.511044 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:42.510529 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" Nov 01 21:54:43.503795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:43.503753 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body= Nov 01 21:54:43.505043 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:43.505008 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" Nov 01 21:54:44.503731 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:44.503698 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body= Nov 01 21:54:44.504125 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:44.503752 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" Nov 01 21:54:45.504159 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:45.504123 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body= Nov 01 21:54:45.504611 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:45.504184 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" Nov 01 21:54:46.162078 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:54:46.162031 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:54:47.290486 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:47.290450 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:47.291265 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:47.291243 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:47.291931 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:47.291912 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" Nov 01 21:54:52.307735 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.307690 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz"] Nov 01 21:54:52.312120 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.312091 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323008 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.322983 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323133 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323039 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323133 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323117 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e570f1-73f4-456e-99de-2128fdbb9d4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323260 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323165 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323324 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323251 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323324 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323297 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323433 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323405 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5js\" (UniqueName: \"kubernetes.io/projected/42e570f1-73f4-456e-99de-2128fdbb9d4e-kube-api-access-fm5js\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323472 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323446 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.323507 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.323472 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.423952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.423913 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.423952 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.423964 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5js\" (UniqueName: \"kubernetes.io/projected/42e570f1-73f4-456e-99de-2128fdbb9d4e-kube-api-access-fm5js\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424292 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424019 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424292 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424046 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424292 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424085 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424583 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424412 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424652 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424591 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e570f1-73f4-456e-99de-2128fdbb9d4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424715 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424651 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424715 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424693 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424839 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424712 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.424839 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.424761 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.425026 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.425004 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.425115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.425056 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.425588 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.425564 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e570f1-73f4-456e-99de-2128fdbb9d4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.427086 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.427053 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.427396 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.427378 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.480682 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.480649 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e570f1-73f4-456e-99de-2128fdbb9d4e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.480850 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.480797 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5js\" (UniqueName: \"kubernetes.io/projected/42e570f1-73f4-456e-99de-2128fdbb9d4e-kube-api-access-fm5js\") pod \"maas-default-gateway-openshift-default-58847f9fc5-x2nzz\" (UID: \"42e570f1-73f4-456e-99de-2128fdbb9d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:52.518566 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.518528 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz"] Nov 01 21:54:52.625273 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:52.625180 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:53.048616 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:53.048586 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz"] Nov 01 21:54:53.049742 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:54:53.049715 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e570f1_73f4_456e_99de_2128fdbb9d4e.slice/crio-af44b2a3e80d056104c513b90cf6803a2977142820011fd422486f0a8781b20b WatchSource:0}: Error finding container af44b2a3e80d056104c513b90cf6803a2977142820011fd422486f0a8781b20b: Status 404 returned error can't find the container with id af44b2a3e80d056104c513b90cf6803a2977142820011fd422486f0a8781b20b Nov 01 21:54:53.051987 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:53.051956 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:54:53.052056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:53.052021 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:54:53.052056 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:53.052052 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:54:53.097795 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:53.097768 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" event={"ID":"42e570f1-73f4-456e-99de-2128fdbb9d4e","Type":"ContainerStarted","Data":"af44b2a3e80d056104c513b90cf6803a2977142820011fd422486f0a8781b20b"} Nov 01 21:54:54.114175 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:54.114031 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" event={"ID":"42e570f1-73f4-456e-99de-2128fdbb9d4e","Type":"ContainerStarted","Data":"c18d8a976a5e509c3392698736ca4157f3b53e8604c1502b8082057de4a0f179"} Nov 01 21:54:54.280347 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:54.280282 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podStartSLOduration=2.280266978 podStartE2EDuration="2.280266978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:54.277187636 +0000 UTC m=+6701.951446891" watchObservedRunningTime="2026-11-01 21:54:54.280266978 +0000 UTC m=+6701.954526234" Nov 01 21:54:54.625474 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:54.625434 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:54:54.627012 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:54.626976 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body= Nov 01 21:54:54.627176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:54.627034 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" Nov 01 21:54:55.626424 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:55.626386 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body= Nov 01 21:54:55.626872 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:55.626445 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" Nov 01 21:54:55.742390 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:55.742353 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:54:56.625888 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:56.625855 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body= Nov 01 21:54:56.626088 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:56.625923 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" Nov 01 21:54:57.625764 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:57.625725 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body= Nov 01 21:54:57.626196 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:57.625794 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" Nov 01 21:54:58.625981 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:58.625945 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body= Nov 01 21:54:58.626428 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:58.626002 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" Nov 01 21:54:59.626513 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:59.626477 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body= Nov 01 21:54:59.626935 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:54:59.626535 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" Nov 01 21:55:00.626657 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:00.626607 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body= Nov 01 21:55:00.627218 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:00.626677 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" Nov 01 21:55:02.519378 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:02.519311 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:55:02.519847 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:02.519796 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:55:02.520608 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:02.520587 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" Nov 01 21:55:07.408231 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.408195 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx"] Nov 01 21:55:07.411866 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.411849 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.477599 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.477569 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbf238c-3422-47b0-92db-30d6379d92ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.477599 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.477602 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.477894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.477631 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.477894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.477805 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.477980 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.477889 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmp6\" (UniqueName: \"kubernetes.io/projected/bdbf238c-3422-47b0-92db-30d6379d92ec-kube-api-access-wkmp6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.477980 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.477942 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.478077 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.477993 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.478121 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.478073 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.478121 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.478092 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.515883 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.515848 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx"] Nov 01 21:55:07.579307 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579267 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579324 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmp6\" (UniqueName: \"kubernetes.io/projected/bdbf238c-3422-47b0-92db-30d6379d92ec-kube-api-access-wkmp6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579407 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579474 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579513 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579543 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579539 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579823 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579580 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbf238c-3422-47b0-92db-30d6379d92ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579916 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579889 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.579916 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579602 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.580037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579938 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.580037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579980 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.580037 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.579991 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.580469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.580448 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.580605 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.580583 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbf238c-3422-47b0-92db-30d6379d92ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.582061 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.582035 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.582160 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.582043 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.636266 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.636233 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbf238c-3422-47b0-92db-30d6379d92ec-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.636448 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.636295 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmp6\" (UniqueName: \"kubernetes.io/projected/bdbf238c-3422-47b0-92db-30d6379d92ec-kube-api-access-wkmp6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ltrbx\" (UID: \"bdbf238c-3422-47b0-92db-30d6379d92ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:07.726385 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:07.726347 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:08.158907 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:08.158829 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx"] Nov 01 21:55:08.196458 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:55:08.196429 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbf238c_3422_47b0_92db_30d6379d92ec.slice/crio-cc2b284683b603cf0845dcf5cb2eb31fbffa31cccb249ad38bfedd22a3b11b98 WatchSource:0}: Error finding container cc2b284683b603cf0845dcf5cb2eb31fbffa31cccb249ad38bfedd22a3b11b98: Status 404 returned error can't find the container with id cc2b284683b603cf0845dcf5cb2eb31fbffa31cccb249ad38bfedd22a3b11b98 Nov 01 21:55:08.198659 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:08.198631 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:55:08.198727 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:08.198699 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:55:08.198765 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:08.198727 5633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 21:55:09.215996 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:09.215959 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" event={"ID":"bdbf238c-3422-47b0-92db-30d6379d92ec","Type":"ContainerStarted","Data":"3fda95af079a31f657069ab9d7768e6e6d9b04a764d712d22457ca532155c834"} Nov 01 21:55:09.216519 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:09.216022 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" event={"ID":"bdbf238c-3422-47b0-92db-30d6379d92ec","Type":"ContainerStarted","Data":"cc2b284683b603cf0845dcf5cb2eb31fbffa31cccb249ad38bfedd22a3b11b98"} Nov 01 21:55:09.726915 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:09.726876 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:09.728414 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:09.728379 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:09.728478 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:09.728440 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 01 21:55:10.727556 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:10.727524 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:10.727962 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:10.727575 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 01 21:55:11.727070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:11.727027 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:11.727251 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:11.727086 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 01 21:55:12.727162 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:12.727118 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:12.727606 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:12.727182 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 01 21:55:13.727022 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:13.726988 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:13.727219 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:13.727039 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 01 21:55:14.727378 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:14.727304 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:14.727809 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:14.727411 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 01 21:55:15.727588 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:15.727542 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:15.728105 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:15.727613 5633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" Nov 01 21:55:16.156734 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:55:16.156639 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:55:17.705428 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:17.705385 5633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:17.705957 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:17.705816 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:17.706873 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:17.706851 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" Nov 01 21:55:17.873491 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:17.873436 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podStartSLOduration=10.873420632 podStartE2EDuration="10.873420632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:09.428989696 +0000 UTC m=+6717.103248950" watchObservedRunningTime="2026-11-01 21:55:17.873420632 +0000 UTC m=+6725.547679886" Nov 01 21:55:46.156942 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:55:46.156896 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:55:49.835664 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:49.835605 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:55.808152 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.808075 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w"] Nov 01 21:55:55.811879 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.811861 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:55.868031 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.867995 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w"] Nov 01 21:55:55.899317 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.899274 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:55.899559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.899362 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhx2\" (UniqueName: \"kubernetes.io/projected/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-kube-api-access-qwhx2\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:55.899559 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.899512 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:55.913579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.913545 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:55:55.913579 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.913554 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:55:55.913719 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.913592 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 21:55:55.913719 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:55.913672 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:55:56.000641 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.000612 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:56.000860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.000716 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:56.000860 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.000790 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhx2\" (UniqueName: \"kubernetes.io/projected/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-kube-api-access-qwhx2\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:56.001456 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.001428 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:56.003176 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.003150 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:56.048419 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.048378 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:55:56.056972 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.056947 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhx2\" (UniqueName: \"kubernetes.io/projected/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-kube-api-access-qwhx2\") pod \"maas-api-revoke-sub-309b3e77-n845w\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:56.125046 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.124946 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:56.497178 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:56.497150 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w"] Nov 01 21:55:56.498498 ip-10-0-128-78 kubenswrapper[5633]: W1101 21:55:56.498471 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776c4af9_83fb_446f_bcde_e8a3abbd0b0f.slice/crio-23e23825edcf1ccd1d5a50eb0865e4626d810d86de2f3c50e49bcebda1a10cf9 WatchSource:0}: Error finding container 23e23825edcf1ccd1d5a50eb0865e4626d810d86de2f3c50e49bcebda1a10cf9: Status 404 returned error can't find the container with id 23e23825edcf1ccd1d5a50eb0865e4626d810d86de2f3c50e49bcebda1a10cf9 Nov 01 21:55:57.496988 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:57.496955 5633 generic.go:356] "Generic (PLEG): container finished" podID="776c4af9-83fb-446f-bcde-e8a3abbd0b0f" containerID="73a07d9e7bf916318acfdc6261a6d32c525bd974a8bb80e9afb7037becccb654" exitCode=0 Nov 01 21:55:57.497408 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:57.497044 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" event={"ID":"776c4af9-83fb-446f-bcde-e8a3abbd0b0f","Type":"ContainerDied","Data":"73a07d9e7bf916318acfdc6261a6d32c525bd974a8bb80e9afb7037becccb654"} Nov 01 21:55:57.497408 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:57.497104 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" event={"ID":"776c4af9-83fb-446f-bcde-e8a3abbd0b0f","Type":"ContainerStarted","Data":"23e23825edcf1ccd1d5a50eb0865e4626d810d86de2f3c50e49bcebda1a10cf9"} Nov 01 21:55:58.647951 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.647924 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:58.731115 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.731083 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-xks-ca\") pod \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " Nov 01 21:55:58.731303 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.731197 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-openshift-ca\") pod \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " Nov 01 21:55:58.731303 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.731232 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-kube-api-access-qwhx2\" (UniqueName: \"kubernetes.io/projected/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-kube-api-access-qwhx2\") pod \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\" (UID: \"776c4af9-83fb-446f-bcde-e8a3abbd0b0f\") " Nov 01 21:55:58.731610 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.731582 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-openshift-ca" pod "776c4af9-83fb-446f-bcde-e8a3abbd0b0f" (UID: "776c4af9-83fb-446f-bcde-e8a3abbd0b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:58.733534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.733506 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-xks-ca" pod "776c4af9-83fb-446f-bcde-e8a3abbd0b0f" (UID: "776c4af9-83fb-446f-bcde-e8a3abbd0b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:58.733534 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.733506 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-kube-api-access-qwhx2" pod "776c4af9-83fb-446f-bcde-e8a3abbd0b0f" (UID: "776c4af9-83fb-446f-bcde-e8a3abbd0b0f"). InnerVolumeSpecName "kube-api-access-qwhx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:58.833023 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.832935 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:55:58.833023 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.832966 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhx2\" (UniqueName: \"kubernetes.io/projected/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-kube-api-access-qwhx2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:55:58.833023 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:58.832976 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776c4af9-83fb-446f-bcde-e8a3abbd0b0f-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:55:59.509469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:59.509442 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" Nov 01 21:55:59.509469 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:59.509458 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w" event={"ID":"776c4af9-83fb-446f-bcde-e8a3abbd0b0f","Type":"ContainerDied","Data":"23e23825edcf1ccd1d5a50eb0865e4626d810d86de2f3c50e49bcebda1a10cf9"} Nov 01 21:55:59.509742 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:55:59.509507 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23e23825edcf1ccd1d5a50eb0865e4626d810d86de2f3c50e49bcebda1a10cf9" Nov 01 21:56:06.578792 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:56:06.578749 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:08.592801 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:56:08.592761 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:56:12.474151 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:56:12.474118 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:16.156591 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:56:16.156545 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:56:46.155674 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:56:46.155632 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:56:56.400322 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:56:56.400278 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:56:59.651439 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:56:59.651394 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:16.157806 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:57:16.157764 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:57:19.800877 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:19.800833 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:19.925558 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:19.925521 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:37.290459 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:37.290363 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:37.751913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:37.751878 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:37.976204 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:37.976170 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:41.494271 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:41.494237 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:41.660849 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:41.660812 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:41.794186 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:41.794093 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:44.681932 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:44.681892 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:44.848276 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:44.848242 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:46.157189 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:57:46.157151 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:57:47.762793 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:47.762755 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:47.896272 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:47.896235 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:48.015223 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:48.015122 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:52.578922 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:52.578878 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:55.765432 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:55.765395 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:57:56.745470 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:56.745432 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:57:57.975751 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:57:57.975717 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:02.137151 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:02.137113 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:06.114306 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:06.114265 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:06.945209 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:06.945174 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:07.085265 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:07.085226 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:07.209489 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:07.209387 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:07.610524 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:07.610479 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:11.834774 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:11.834737 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:13.439138 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:13.439104 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:16.157158 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:58:16.157120 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:58:16.473070 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:16.473035 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:21.174562 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:21.174522 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:25.871664 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:25.871626 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:30.562189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:30.562154 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:35.277362 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:35.277293 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:39.893189 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:39.893151 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:44.597103 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:44.597063 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:46.156047 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:58:46.156010 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:58:49.271269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:49.271220 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:53.980894 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:53.980804 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:56.163781 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:56.163741 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:56.298400 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:56.298364 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:57.081230 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:57.081194 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:58:58.675479 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:58:58.675444 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:03.376087 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:03.376050 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:08.066464 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:08.066426 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:12.771009 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:12.770970 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:16.157681 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:59:16.157636 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:59:17.479071 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:17.479026 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:19.056737 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:19.056696 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc"] Nov 01 21:59:19.105732 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:19.105689 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca1f3380-kmbrc"] Nov 01 21:59:19.310264 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:19.310162 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:19.428074 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:19.428029 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:20.466519 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:20.466482 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb" path="/var/lib/kubelet/pods/d034cc96-4a6d-48e9-870d-a1dd7a6cd7eb/volumes" Nov 01 21:59:22.171396 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:22.171357 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:26.871731 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:26.871670 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:31.583276 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:31.583235 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:36.277459 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:36.277420 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:38.273837 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:38.273801 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:41.046529 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:41.046475 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:44.232944 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:44.232906 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:44.405985 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:44.405941 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:44.532537 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:44.532440 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:45.853675 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:45.853641 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:46.156686 ip-10-0-128-78 kubenswrapper[5633]: E1101 21:59:46.156601 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 21:59:49.940269 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:49.940223 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:50.120463 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:50.120429 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:54.579607 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:54.579568 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 21:59:55.580913 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:55.580868 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:55.788768 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:55.788724 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:57.089698 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:57.089654 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:57.423551 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:57.423455 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 21:59:57.693945 ip-10-0-128-78 kubenswrapper[5633]: I1101 21:59:57.693853 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:00.164149 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.164118 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7"] Nov 01 22:00:00.164657 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.164643 5633 cpu_manager.go:414] "containerMap: removing container" podUID="776c4af9-83fb-446f-bcde-e8a3abbd0b0f" containerName="revoke-keys" Nov 01 22:00:00.164704 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.164659 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776c4af9-83fb-446f-bcde-e8a3abbd0b0f" containerName="revoke-keys" Nov 01 22:00:00.164745 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.164722 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="776c4af9-83fb-446f-bcde-e8a3abbd0b0f" containerName="revoke-keys" Nov 01 22:00:00.168077 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.168061 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.224200 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.224164 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7"] Nov 01 22:00:00.267170 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.267131 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da7f19-4147-483b-b35e-719d1826b1d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892840-zwnr7\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.267379 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.267290 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwq2\" (UniqueName: \"kubernetes.io/projected/d0da7f19-4147-483b-b35e-719d1826b1d5-kube-api-access-xvwq2\") pod \"maas-api-key-cleanup-29892840-zwnr7\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.269538 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.269516 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:00:00.269601 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.269525 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:00:00.269601 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.269554 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:00:00.368876 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.368835 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwq2\" (UniqueName: \"kubernetes.io/projected/d0da7f19-4147-483b-b35e-719d1826b1d5-kube-api-access-xvwq2\") pod \"maas-api-key-cleanup-29892840-zwnr7\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.369076 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.368908 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da7f19-4147-483b-b35e-719d1826b1d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892840-zwnr7\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.369612 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.369591 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da7f19-4147-483b-b35e-719d1826b1d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892840-zwnr7\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.425264 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.425184 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwq2\" (UniqueName: \"kubernetes.io/projected/d0da7f19-4147-483b-b35e-719d1826b1d5-kube-api-access-xvwq2\") pod \"maas-api-key-cleanup-29892840-zwnr7\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.481666 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.481633 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:00.853935 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:00.853910 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7"] Nov 01 22:00:00.855398 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:00:00.855321 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0da7f19_4147_483b_b35e_719d1826b1d5.slice/crio-aa4a656e17daeb43d227249a50dd7e382b070b826209ef248b4b83371fec456d WatchSource:0}: Error finding container aa4a656e17daeb43d227249a50dd7e382b070b826209ef248b4b83371fec456d: Status 404 returned error can't find the container with id aa4a656e17daeb43d227249a50dd7e382b070b826209ef248b4b83371fec456d Nov 01 22:00:01.862227 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:01.862184 5633 generic.go:356] "Generic (PLEG): container finished" podID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerID="ea8c2b4e00b19e63fe9a15ac1b7784b06f0f97912ca6d1b48a904d0fd141a3d9" exitCode=60 Nov 01 22:00:01.862671 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:01.862277 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" event={"ID":"d0da7f19-4147-483b-b35e-719d1826b1d5","Type":"ContainerDied","Data":"ea8c2b4e00b19e63fe9a15ac1b7784b06f0f97912ca6d1b48a904d0fd141a3d9"} Nov 01 22:00:01.862671 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:01.862351 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" event={"ID":"d0da7f19-4147-483b-b35e-719d1826b1d5","Type":"ContainerStarted","Data":"aa4a656e17daeb43d227249a50dd7e382b070b826209ef248b4b83371fec456d"} Nov 01 22:00:01.862671 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:01.862534 5633 scope.go:122] "RemoveContainer" containerID="ea8c2b4e00b19e63fe9a15ac1b7784b06f0f97912ca6d1b48a904d0fd141a3d9" Nov 01 22:00:02.871250 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:02.871214 5633 generic.go:356] "Generic (PLEG): container finished" podID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerID="4016c1126c2a92b606956445d6790a83477daf7c067b213a40eea13064909244" exitCode=60 Nov 01 22:00:02.871736 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:02.871301 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" event={"ID":"d0da7f19-4147-483b-b35e-719d1826b1d5","Type":"ContainerDied","Data":"4016c1126c2a92b606956445d6790a83477daf7c067b213a40eea13064909244"} Nov 01 22:00:02.871736 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:02.871391 5633 scope.go:122] "RemoveContainer" containerID="ea8c2b4e00b19e63fe9a15ac1b7784b06f0f97912ca6d1b48a904d0fd141a3d9" Nov 01 22:00:02.871736 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:02.871659 5633 scope.go:122] "RemoveContainer" containerID="4016c1126c2a92b606956445d6790a83477daf7c067b213a40eea13064909244" Nov 01 22:00:02.871984 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:00:02.871954 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892840-zwnr7_odh-ai-gateway-infra(d0da7f19-4147-483b-b35e-719d1826b1d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" Nov 01 22:00:03.880139 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:03.880109 5633 scope.go:122] "RemoveContainer" containerID="4016c1126c2a92b606956445d6790a83477daf7c067b213a40eea13064909244" Nov 01 22:00:03.880586 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:00:03.880386 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892840-zwnr7_odh-ai-gateway-infra(d0da7f19-4147-483b-b35e-719d1826b1d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" Nov 01 22:00:16.156456 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:00:16.156416 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:00:16.459978 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:16.459884 5633 scope.go:122] "RemoveContainer" containerID="4016c1126c2a92b606956445d6790a83477daf7c067b213a40eea13064909244" Nov 01 22:00:16.961111 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:16.961073 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" event={"ID":"d0da7f19-4147-483b-b35e-719d1826b1d5","Type":"ContainerStarted","Data":"7d2be449d512a617d06bfba88986a43625206ae8781b6d2004d249c775369b6c"} Nov 01 22:00:17.074588 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:17.074480 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" podStartSLOduration=17.074466291 podStartE2EDuration="17.074466291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:17.072824565 +0000 UTC m=+7024.747083823" watchObservedRunningTime="2026-11-01 22:00:17.074466291 +0000 UTC m=+7024.748725545" Nov 01 22:00:17.969155 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:17.969122 5633 generic.go:356] "Generic (PLEG): container finished" podID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerID="7d2be449d512a617d06bfba88986a43625206ae8781b6d2004d249c775369b6c" exitCode=60 Nov 01 22:00:17.969638 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:17.969193 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" event={"ID":"d0da7f19-4147-483b-b35e-719d1826b1d5","Type":"ContainerDied","Data":"7d2be449d512a617d06bfba88986a43625206ae8781b6d2004d249c775369b6c"} Nov 01 22:00:17.969638 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:17.969258 5633 scope.go:122] "RemoveContainer" containerID="4016c1126c2a92b606956445d6790a83477daf7c067b213a40eea13064909244" Nov 01 22:00:17.969638 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:17.969588 5633 scope.go:122] "RemoveContainer" containerID="7d2be449d512a617d06bfba88986a43625206ae8781b6d2004d249c775369b6c" Nov 01 22:00:17.969841 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:00:17.969821 5633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892840-zwnr7_odh-ai-gateway-infra(d0da7f19-4147-483b-b35e-719d1826b1d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" Nov 01 22:00:18.082676 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:18.082641 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7"] Nov 01 22:00:19.126074 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.126049 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:19.282643 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.282565 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0da7f19-4147-483b-b35e-719d1826b1d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da7f19-4147-483b-b35e-719d1826b1d5-maas-api-service-ca\") pod \"d0da7f19-4147-483b-b35e-719d1826b1d5\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " Nov 01 22:00:19.282796 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.282672 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0da7f19-4147-483b-b35e-719d1826b1d5-kube-api-access-xvwq2\" (UniqueName: \"kubernetes.io/projected/d0da7f19-4147-483b-b35e-719d1826b1d5-kube-api-access-xvwq2\") pod \"d0da7f19-4147-483b-b35e-719d1826b1d5\" (UID: \"d0da7f19-4147-483b-b35e-719d1826b1d5\") " Nov 01 22:00:19.282932 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.282911 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0da7f19-4147-483b-b35e-719d1826b1d5-maas-api-service-ca" pod "d0da7f19-4147-483b-b35e-719d1826b1d5" (UID: "d0da7f19-4147-483b-b35e-719d1826b1d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:19.284979 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.284959 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0da7f19-4147-483b-b35e-719d1826b1d5-kube-api-access-xvwq2" pod "d0da7f19-4147-483b-b35e-719d1826b1d5" (UID: "d0da7f19-4147-483b-b35e-719d1826b1d5"). InnerVolumeSpecName "kube-api-access-xvwq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:19.384189 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.384150 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da7f19-4147-483b-b35e-719d1826b1d5-maas-api-service-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.384189 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.384179 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwq2\" (UniqueName: \"kubernetes.io/projected/d0da7f19-4147-483b-b35e-719d1826b1d5-kube-api-access-xvwq2\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:00:19.489192 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.489156 5633 scope.go:122] "RemoveContainer" containerID="e31ef33aa498df070a9b86c1eef0968a6a4ae7cb089adecaedb855a74dbbc287" Nov 01 22:00:19.983915 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.983886 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" Nov 01 22:00:19.984118 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.983907 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7" event={"ID":"d0da7f19-4147-483b-b35e-719d1826b1d5","Type":"ContainerDied","Data":"aa4a656e17daeb43d227249a50dd7e382b070b826209ef248b4b83371fec456d"} Nov 01 22:00:19.984118 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:19.983974 5633 scope.go:122] "RemoveContainer" containerID="7d2be449d512a617d06bfba88986a43625206ae8781b6d2004d249c775369b6c" Nov 01 22:00:20.145959 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:20.145917 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7"] Nov 01 22:00:20.195052 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:20.195016 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892840-zwnr7"] Nov 01 22:00:20.466215 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:20.466184 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" path="/var/lib/kubelet/pods/d0da7f19-4147-483b-b35e-719d1826b1d5/volumes" Nov 01 22:00:31.694240 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:31.694153 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:00:44.988659 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.988619 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv"] Nov 01 22:00:44.989099 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.989087 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:44.989152 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.989098 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:44.989152 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.989119 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:44.989152 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.989124 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:44.989274 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.989209 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:44.989274 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.989226 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:44.992486 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:44.992470 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.016540 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.016511 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.016691 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.016546 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.016691 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.016578 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95hd\" (UniqueName: \"kubernetes.io/projected/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-kube-api-access-p95hd\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.049560 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.049521 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv"] Nov 01 22:00:45.095386 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.095312 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:00:45.095386 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.095321 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:00:45.095386 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.095390 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:00:45.095656 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.095437 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:00:45.117143 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.117110 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.117143 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.117146 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.117448 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.117283 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95hd\" (UniqueName: \"kubernetes.io/projected/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-kube-api-access-p95hd\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.117885 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.117862 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.119741 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.119725 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.174694 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.174662 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95hd\" (UniqueName: \"kubernetes.io/projected/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-kube-api-access-p95hd\") pod \"maas-api-revoke-sub-96a468d0-4xhvv\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.308127 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.308043 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:45.685441 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:45.685413 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv"] Nov 01 22:00:45.686828 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:00:45.686800 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa8a534_7cde_44c2_99c2_ff69a0bdc821.slice/crio-96f55bbe0683b1b0fc7022b3bd23a7814951f3753b97892fb846884874b6cde7 WatchSource:0}: Error finding container 96f55bbe0683b1b0fc7022b3bd23a7814951f3753b97892fb846884874b6cde7: Status 404 returned error can't find the container with id 96f55bbe0683b1b0fc7022b3bd23a7814951f3753b97892fb846884874b6cde7 Nov 01 22:00:46.156989 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:00:46.156952 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:00:46.212824 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:46.212789 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" event={"ID":"ffa8a534-7cde-44c2-99c2-ff69a0bdc821","Type":"ContainerStarted","Data":"bec303896bd5f8ba4e4811717380cede39368dd1d2e12c012cc2c834a83dd942"} Nov 01 22:00:46.213017 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:46.212837 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" event={"ID":"ffa8a534-7cde-44c2-99c2-ff69a0bdc821","Type":"ContainerStarted","Data":"96f55bbe0683b1b0fc7022b3bd23a7814951f3753b97892fb846884874b6cde7"} Nov 01 22:00:46.327652 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:46.327541 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" podStartSLOduration=2.327524652 podStartE2EDuration="2.327524652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:46.324978662 +0000 UTC m=+7053.999237921" watchObservedRunningTime="2026-11-01 22:00:46.327524652 +0000 UTC m=+7054.001783906" Nov 01 22:00:47.220190 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:47.220160 5633 generic.go:356] "Generic (PLEG): container finished" podID="ffa8a534-7cde-44c2-99c2-ff69a0bdc821" containerID="bec303896bd5f8ba4e4811717380cede39368dd1d2e12c012cc2c834a83dd942" exitCode=0 Nov 01 22:00:47.220634 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:47.220296 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" event={"ID":"ffa8a534-7cde-44c2-99c2-ff69a0bdc821","Type":"ContainerDied","Data":"bec303896bd5f8ba4e4811717380cede39368dd1d2e12c012cc2c834a83dd942"} Nov 01 22:00:48.383997 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.383972 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:48.451532 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.451496 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-openshift-ca\") pod \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " Nov 01 22:00:48.451711 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.451642 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-kube-api-access-p95hd\" (UniqueName: \"kubernetes.io/projected/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-kube-api-access-p95hd\") pod \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " Nov 01 22:00:48.451756 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.451706 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-xks-ca\") pod \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\" (UID: \"ffa8a534-7cde-44c2-99c2-ff69a0bdc821\") " Nov 01 22:00:48.451942 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.451921 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-openshift-ca" pod "ffa8a534-7cde-44c2-99c2-ff69a0bdc821" (UID: "ffa8a534-7cde-44c2-99c2-ff69a0bdc821"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:48.452081 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.452062 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:00:48.453943 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.453909 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-kube-api-access-p95hd" pod "ffa8a534-7cde-44c2-99c2-ff69a0bdc821" (UID: "ffa8a534-7cde-44c2-99c2-ff69a0bdc821"). InnerVolumeSpecName "kube-api-access-p95hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:48.454046 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.453917 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-xks-ca" pod "ffa8a534-7cde-44c2-99c2-ff69a0bdc821" (UID: "ffa8a534-7cde-44c2-99c2-ff69a0bdc821"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:00:48.553440 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.553307 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95hd\" (UniqueName: \"kubernetes.io/projected/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-kube-api-access-p95hd\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:00:48.553440 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:48.553367 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa8a534-7cde-44c2-99c2-ff69a0bdc821-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:00:49.232179 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:49.232148 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" Nov 01 22:00:49.232420 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:49.232182 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv" event={"ID":"ffa8a534-7cde-44c2-99c2-ff69a0bdc821","Type":"ContainerDied","Data":"96f55bbe0683b1b0fc7022b3bd23a7814951f3753b97892fb846884874b6cde7"} Nov 01 22:00:49.232420 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:49.232276 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f55bbe0683b1b0fc7022b3bd23a7814951f3753b97892fb846884874b6cde7" Nov 01 22:00:55.714384 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.714349 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t"] Nov 01 22:00:55.714818 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.714793 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ffa8a534-7cde-44c2-99c2-ff69a0bdc821" containerName="revoke-keys" Nov 01 22:00:55.714818 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.714804 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa8a534-7cde-44c2-99c2-ff69a0bdc821" containerName="revoke-keys" Nov 01 22:00:55.714916 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.714825 5633 cpu_manager.go:414] "containerMap: removing container" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:55.714916 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.714830 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:55.714916 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.714912 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa8a534-7cde-44c2-99c2-ff69a0bdc821" containerName="revoke-keys" Nov 01 22:00:55.715029 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.714920 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0da7f19-4147-483b-b35e-719d1826b1d5" containerName="cleanup" Nov 01 22:00:55.718645 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.718623 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.774514 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.774477 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t"] Nov 01 22:00:55.819187 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.819149 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:00:55.819187 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.819165 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:00:55.819454 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.819252 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:00:55.820492 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.820468 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:00:55.827069 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.827041 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.827196 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.827112 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.827196 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.827170 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4xp\" (UniqueName: \"kubernetes.io/projected/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-kube-api-access-rx4xp\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.928482 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.928447 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.928672 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.928512 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.928672 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.928532 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4xp\" (UniqueName: \"kubernetes.io/projected/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-kube-api-access-rx4xp\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.929113 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.929080 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.931043 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.931018 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:55.984815 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:55.984738 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4xp\" (UniqueName: \"kubernetes.io/projected/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-kube-api-access-rx4xp\") pod \"maas-api-revoke-sub-a14a6aad-qns8t\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:56.033100 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:56.033061 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:56.436689 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:56.436660 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t"] Nov 01 22:00:56.437480 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:00:56.437441 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc4afd2_1b18_4324_b9ba_a0bfd8b5ba03.slice/crio-1c06d4420a4fe8fd81c17ce262d77a457a1929fd262c032c37d321c7d7811098 WatchSource:0}: Error finding container 1c06d4420a4fe8fd81c17ce262d77a457a1929fd262c032c37d321c7d7811098: Status 404 returned error can't find the container with id 1c06d4420a4fe8fd81c17ce262d77a457a1929fd262c032c37d321c7d7811098 Nov 01 22:00:57.277012 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:57.276975 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" event={"ID":"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03","Type":"ContainerStarted","Data":"ddb9dd13e8001093db2d263b5759dba26bd0f7bbf4e99a3f8a825edc108f4e22"} Nov 01 22:00:57.277467 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:57.277029 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" event={"ID":"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03","Type":"ContainerStarted","Data":"1c06d4420a4fe8fd81c17ce262d77a457a1929fd262c032c37d321c7d7811098"} Nov 01 22:00:57.390835 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:57.390774 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" podStartSLOduration=2.390758535 podStartE2EDuration="2.390758535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:57.388692958 +0000 UTC m=+7065.062952225" watchObservedRunningTime="2026-11-01 22:00:57.390758535 +0000 UTC m=+7065.065017790" Nov 01 22:00:57.813836 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:57.813800 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:00:58.285038 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:58.285005 5633 generic.go:356] "Generic (PLEG): container finished" podID="bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" containerID="ddb9dd13e8001093db2d263b5759dba26bd0f7bbf4e99a3f8a825edc108f4e22" exitCode=0 Nov 01 22:00:58.285489 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:58.285052 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" event={"ID":"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03","Type":"ContainerDied","Data":"ddb9dd13e8001093db2d263b5759dba26bd0f7bbf4e99a3f8a825edc108f4e22"} Nov 01 22:00:59.043306 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.043272 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w"] Nov 01 22:00:59.092321 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.092280 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309b3e77-n845w"] Nov 01 22:00:59.431308 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.431279 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:00:59.567600 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.567557 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-openshift-ca\") pod \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " Nov 01 22:00:59.567802 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.567683 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-xks-ca\") pod \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " Nov 01 22:00:59.567883 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.567864 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-kube-api-access-rx4xp\" (UniqueName: \"kubernetes.io/projected/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-kube-api-access-rx4xp\") pod \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\" (UID: \"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03\") " Nov 01 22:00:59.567946 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.567907 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-openshift-ca" pod "bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" (UID: "bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:59.568301 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.568275 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:00:59.570216 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.570187 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-xks-ca" pod "bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" (UID: "bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:00:59.570401 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.570384 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-kube-api-access-rx4xp" pod "bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" (UID: "bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03"). InnerVolumeSpecName "kube-api-access-rx4xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:59.669261 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.669176 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4xp\" (UniqueName: \"kubernetes.io/projected/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-kube-api-access-rx4xp\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:00:59.669261 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:00:59.669208 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:01:00.297597 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:00.297554 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" event={"ID":"bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03","Type":"ContainerDied","Data":"1c06d4420a4fe8fd81c17ce262d77a457a1929fd262c032c37d321c7d7811098"} Nov 01 22:01:00.297597 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:00.297583 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t" Nov 01 22:01:00.297833 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:00.297618 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c06d4420a4fe8fd81c17ce262d77a457a1929fd262c032c37d321c7d7811098" Nov 01 22:01:00.466279 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:00.466246 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776c4af9-83fb-446f-bcde-e8a3abbd0b0f" path="/var/lib/kubelet/pods/776c4af9-83fb-446f-bcde-e8a3abbd0b0f/volumes" Nov 01 22:01:02.752120 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:02.752072 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:08.053214 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:08.053183 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:08.189637 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:08.189602 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:01:08.491825 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:08.491791 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:01:16.157801 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:01:16.157742 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:01:18.340564 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:18.340512 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:01:18.341192 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:18.341157 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:18.570529 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:18.570499 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:01:18.570779 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:18.570756 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:18.752574 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:18.752521 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:01:18.753288 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:18.753250 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:21.368607 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:21.368564 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:01:31.000467 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.000431 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg"] Nov 01 22:01:31.000896 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.000889 5633 cpu_manager.go:414] "containerMap: removing container" podUID="bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" containerName="revoke-keys" Nov 01 22:01:31.000950 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.000900 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" containerName="revoke-keys" Nov 01 22:01:31.000996 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.000989 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" containerName="revoke-keys" Nov 01 22:01:31.006072 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.006044 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.060877 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.060843 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg"] Nov 01 22:01:31.105302 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.105258 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:01:31.105514 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.105307 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:01:31.107104 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.107081 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:01:31.107104 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.107081 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:01:31.107519 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.107502 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.107573 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.107542 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.107632 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.107619 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krvwr\" (UniqueName: \"kubernetes.io/projected/4c9cb3f6-2c39-431f-963e-08b4afeaf953-kube-api-access-krvwr\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.208319 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.208281 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.208576 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.208425 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krvwr\" (UniqueName: \"kubernetes.io/projected/4c9cb3f6-2c39-431f-963e-08b4afeaf953-kube-api-access-krvwr\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.208576 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.208544 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.208987 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.208953 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.211067 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.211048 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.264929 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.264856 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krvwr\" (UniqueName: \"kubernetes.io/projected/4c9cb3f6-2c39-431f-963e-08b4afeaf953-kube-api-access-krvwr\") pod \"maas-api-revoke-sub-750cf080-4mnqg\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.320728 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.320687 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:31.695196 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:31.695164 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg"] Nov 01 22:01:31.696735 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:01:31.696699 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9cb3f6_2c39_431f_963e_08b4afeaf953.slice/crio-5fed3415898abf5d7022132f3a2cfa3803e0af92f84a09d267aaf851013d5fe0 WatchSource:0}: Error finding container 5fed3415898abf5d7022132f3a2cfa3803e0af92f84a09d267aaf851013d5fe0: Status 404 returned error can't find the container with id 5fed3415898abf5d7022132f3a2cfa3803e0af92f84a09d267aaf851013d5fe0 Nov 01 22:01:32.506800 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:32.506767 5633 generic.go:356] "Generic (PLEG): container finished" podID="4c9cb3f6-2c39-431f-963e-08b4afeaf953" containerID="2b92f12fa09a5097072af3e206ce628e935371b3f7d9e605162756506bf471e1" exitCode=0 Nov 01 22:01:32.507131 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:32.506848 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" event={"ID":"4c9cb3f6-2c39-431f-963e-08b4afeaf953","Type":"ContainerDied","Data":"2b92f12fa09a5097072af3e206ce628e935371b3f7d9e605162756506bf471e1"} Nov 01 22:01:32.507131 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:32.506902 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" event={"ID":"4c9cb3f6-2c39-431f-963e-08b4afeaf953","Type":"ContainerStarted","Data":"5fed3415898abf5d7022132f3a2cfa3803e0af92f84a09d267aaf851013d5fe0"} Nov 01 22:01:33.661214 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.661186 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:33.736961 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.736925 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-openshift-ca\") pod \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " Nov 01 22:01:33.737145 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.737041 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9cb3f6-2c39-431f-963e-08b4afeaf953-kube-api-access-krvwr\" (UniqueName: \"kubernetes.io/projected/4c9cb3f6-2c39-431f-963e-08b4afeaf953-kube-api-access-krvwr\") pod \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " Nov 01 22:01:33.737206 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.737184 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-xks-ca\") pod \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\" (UID: \"4c9cb3f6-2c39-431f-963e-08b4afeaf953\") " Nov 01 22:01:33.737373 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.737354 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-openshift-ca" pod "4c9cb3f6-2c39-431f-963e-08b4afeaf953" (UID: "4c9cb3f6-2c39-431f-963e-08b4afeaf953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:01:33.737568 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.737546 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:01:33.739450 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.739420 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-xks-ca" pod "4c9cb3f6-2c39-431f-963e-08b4afeaf953" (UID: "4c9cb3f6-2c39-431f-963e-08b4afeaf953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:01:33.739573 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.739456 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9cb3f6-2c39-431f-963e-08b4afeaf953-kube-api-access-krvwr" pod "4c9cb3f6-2c39-431f-963e-08b4afeaf953" (UID: "4c9cb3f6-2c39-431f-963e-08b4afeaf953"). InnerVolumeSpecName "kube-api-access-krvwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:01:33.838484 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.838389 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krvwr\" (UniqueName: \"kubernetes.io/projected/4c9cb3f6-2c39-431f-963e-08b4afeaf953-kube-api-access-krvwr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:01:33.838484 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:33.838426 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9cb3f6-2c39-431f-963e-08b4afeaf953-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:01:34.518564 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:34.518524 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" event={"ID":"4c9cb3f6-2c39-431f-963e-08b4afeaf953","Type":"ContainerDied","Data":"5fed3415898abf5d7022132f3a2cfa3803e0af92f84a09d267aaf851013d5fe0"} Nov 01 22:01:34.518759 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:34.518577 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fed3415898abf5d7022132f3a2cfa3803e0af92f84a09d267aaf851013d5fe0" Nov 01 22:01:34.518759 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:34.518580 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg" Nov 01 22:01:43.869844 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:43.869803 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:01:46.156657 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:01:46.156620 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:01:55.555306 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:55.555220 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:01:58.269916 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:01:58.269881 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:12.318190 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.318157 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq"] Nov 01 22:02:12.318705 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.318690 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4c9cb3f6-2c39-431f-963e-08b4afeaf953" containerName="revoke-keys" Nov 01 22:02:12.318755 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.318706 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9cb3f6-2c39-431f-963e-08b4afeaf953" containerName="revoke-keys" Nov 01 22:02:12.318796 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.318776 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9cb3f6-2c39-431f-963e-08b4afeaf953" containerName="revoke-keys" Nov 01 22:02:12.321830 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.321813 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.377750 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.377714 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq"] Nov 01 22:02:12.423171 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.423133 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:02:12.423401 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.423224 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:02:12.423401 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.423232 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:02:12.423401 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.423287 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:02:12.423759 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.423739 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgdn\" (UniqueName: \"kubernetes.io/projected/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-kube-api-access-7mgdn\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.423858 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.423842 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.423910 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.423897 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.525232 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.525194 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgdn\" (UniqueName: \"kubernetes.io/projected/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-kube-api-access-7mgdn\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.525464 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.525273 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.525464 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.525345 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.576386 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.576281 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:02:12.576574 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.576440 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:02:12.586895 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.586862 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.588669 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.588637 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.629515 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.629486 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:02:12.642367 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.642316 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgdn\" (UniqueName: \"kubernetes.io/projected/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-kube-api-access-7mgdn\") pod \"maas-api-revoke-sub-75a6d999-828tq\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:12.985858 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.985831 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:02:12.995953 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:12.995921 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:13.370323 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:13.370296 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq"] Nov 01 22:02:13.371779 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:02:13.371747 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0578d5_b13f_4b5f_b6c5_88876a70bc98.slice/crio-e88653ddbc6a18db321202fa81f0df9c34f1d5f291e13f54b8cbe0881b3d85ad WatchSource:0}: Error finding container e88653ddbc6a18db321202fa81f0df9c34f1d5f291e13f54b8cbe0881b3d85ad: Status 404 returned error can't find the container with id e88653ddbc6a18db321202fa81f0df9c34f1d5f291e13f54b8cbe0881b3d85ad Nov 01 22:02:13.769661 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:13.769623 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" event={"ID":"4d0578d5-b13f-4b5f-b6c5-88876a70bc98","Type":"ContainerStarted","Data":"c713ccb74d88049d68f5127b20c3641658571bbf1260409e55246f90f7ffb6ec"} Nov 01 22:02:13.769855 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:13.769686 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" event={"ID":"4d0578d5-b13f-4b5f-b6c5-88876a70bc98","Type":"ContainerStarted","Data":"e88653ddbc6a18db321202fa81f0df9c34f1d5f291e13f54b8cbe0881b3d85ad"} Nov 01 22:02:13.881894 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:13.881832 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" podStartSLOduration=1.881816627 podStartE2EDuration="1.881816627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:02:13.880577949 +0000 UTC m=+7141.554837205" watchObservedRunningTime="2026-11-01 22:02:13.881816627 +0000 UTC m=+7141.556075953" Nov 01 22:02:14.776937 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:14.776899 5633 generic.go:356] "Generic (PLEG): container finished" podID="4d0578d5-b13f-4b5f-b6c5-88876a70bc98" containerID="c713ccb74d88049d68f5127b20c3641658571bbf1260409e55246f90f7ffb6ec" exitCode=0 Nov 01 22:02:14.777375 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:14.776991 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" event={"ID":"4d0578d5-b13f-4b5f-b6c5-88876a70bc98","Type":"ContainerDied","Data":"c713ccb74d88049d68f5127b20c3641658571bbf1260409e55246f90f7ffb6ec"} Nov 01 22:02:15.922406 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:15.922381 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:16.065700 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.065614 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-openshift-ca\") pod \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " Nov 01 22:02:16.065869 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.065810 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-xks-ca\") pod \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " Nov 01 22:02:16.065869 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.065841 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-kube-api-access-7mgdn\" (UniqueName: \"kubernetes.io/projected/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-kube-api-access-7mgdn\") pod \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\" (UID: \"4d0578d5-b13f-4b5f-b6c5-88876a70bc98\") " Nov 01 22:02:16.066089 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.066058 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-openshift-ca" pod "4d0578d5-b13f-4b5f-b6c5-88876a70bc98" (UID: "4d0578d5-b13f-4b5f-b6c5-88876a70bc98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:02:16.068157 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.068129 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-xks-ca" pod "4d0578d5-b13f-4b5f-b6c5-88876a70bc98" (UID: "4d0578d5-b13f-4b5f-b6c5-88876a70bc98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:02:16.068289 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.068157 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-kube-api-access-7mgdn" pod "4d0578d5-b13f-4b5f-b6c5-88876a70bc98" (UID: "4d0578d5-b13f-4b5f-b6c5-88876a70bc98"). InnerVolumeSpecName "kube-api-access-7mgdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:02:16.156286 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:02:16.156249 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:02:16.166928 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.166904 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:02:16.167010 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.166933 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mgdn\" (UniqueName: \"kubernetes.io/projected/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-kube-api-access-7mgdn\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:02:16.167010 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.166944 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d0578d5-b13f-4b5f-b6c5-88876a70bc98-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:02:16.789595 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.789508 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" event={"ID":"4d0578d5-b13f-4b5f-b6c5-88876a70bc98","Type":"ContainerDied","Data":"e88653ddbc6a18db321202fa81f0df9c34f1d5f291e13f54b8cbe0881b3d85ad"} Nov 01 22:02:16.789595 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.789587 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88653ddbc6a18db321202fa81f0df9c34f1d5f291e13f54b8cbe0881b3d85ad" Nov 01 22:02:16.789595 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:16.789521 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq" Nov 01 22:02:19.526447 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:19.526418 5633 scope.go:122] "RemoveContainer" containerID="73a07d9e7bf916318acfdc6261a6d32c525bd974a8bb80e9afb7037becccb654" Nov 01 22:02:23.091579 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.091536 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w"] Nov 01 22:02:23.092209 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.092187 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4d0578d5-b13f-4b5f-b6c5-88876a70bc98" containerName="revoke-keys" Nov 01 22:02:23.092430 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.092408 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0578d5-b13f-4b5f-b6c5-88876a70bc98" containerName="revoke-keys" Nov 01 22:02:23.092606 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.092591 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0578d5-b13f-4b5f-b6c5-88876a70bc98" containerName="revoke-keys" Nov 01 22:02:23.097602 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.097576 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.151720 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.151683 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w"] Nov 01 22:02:23.198783 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.198747 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:02:23.198783 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.198777 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:02:23.199031 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.198777 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:02:23.246473 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.246167 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:02:23.249825 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.249799 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.249989 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.249870 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfv22\" (UniqueName: \"kubernetes.io/projected/6c10247a-e55b-420e-9829-c9584b5afa3e-kube-api-access-pfv22\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.249989 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.249975 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.350834 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.350744 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.350834 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.350819 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfv22\" (UniqueName: \"kubernetes.io/projected/6c10247a-e55b-420e-9829-c9584b5afa3e-kube-api-access-pfv22\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.351042 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.350881 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.351469 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.351436 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.353450 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.353430 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.407228 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.407192 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfv22\" (UniqueName: \"kubernetes.io/projected/6c10247a-e55b-420e-9829-c9584b5afa3e-kube-api-access-pfv22\") pod \"maas-api-revoke-sub-887d0aa3-plr8w\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.412899 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.412875 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:23.785883 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.785855 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w"] Nov 01 22:02:23.787443 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:02:23.787410 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c10247a_e55b_420e_9829_c9584b5afa3e.slice/crio-aa4af97456c9a0afd233bf9b3f8fb39248464ab8c33c1c98b5d56bcbcc4de16c WatchSource:0}: Error finding container aa4af97456c9a0afd233bf9b3f8fb39248464ab8c33c1c98b5d56bcbcc4de16c: Status 404 returned error can't find the container with id aa4af97456c9a0afd233bf9b3f8fb39248464ab8c33c1c98b5d56bcbcc4de16c Nov 01 22:02:23.829818 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:23.829783 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" event={"ID":"6c10247a-e55b-420e-9829-c9584b5afa3e","Type":"ContainerStarted","Data":"aa4af97456c9a0afd233bf9b3f8fb39248464ab8c33c1c98b5d56bcbcc4de16c"} Nov 01 22:02:24.837253 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:24.837217 5633 generic.go:356] "Generic (PLEG): container finished" podID="6c10247a-e55b-420e-9829-c9584b5afa3e" containerID="a8fac4eac12e2086f0ede47dc812686739a0302a25b89bc78bb9fe7d54daad5a" exitCode=0 Nov 01 22:02:24.837741 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:24.837272 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" event={"ID":"6c10247a-e55b-420e-9829-c9584b5afa3e","Type":"ContainerDied","Data":"a8fac4eac12e2086f0ede47dc812686739a0302a25b89bc78bb9fe7d54daad5a"} Nov 01 22:02:25.105462 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:25.105419 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:02:25.991604 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:25.991572 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:26.079907 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.079873 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-xks-ca\") pod \"6c10247a-e55b-420e-9829-c9584b5afa3e\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " Nov 01 22:02:26.080106 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.079941 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-openshift-ca\") pod \"6c10247a-e55b-420e-9829-c9584b5afa3e\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " Nov 01 22:02:26.080106 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.080022 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c10247a-e55b-420e-9829-c9584b5afa3e-kube-api-access-pfv22\" (UniqueName: \"kubernetes.io/projected/6c10247a-e55b-420e-9829-c9584b5afa3e-kube-api-access-pfv22\") pod \"6c10247a-e55b-420e-9829-c9584b5afa3e\" (UID: \"6c10247a-e55b-420e-9829-c9584b5afa3e\") " Nov 01 22:02:26.080372 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.080317 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-openshift-ca" pod "6c10247a-e55b-420e-9829-c9584b5afa3e" (UID: "6c10247a-e55b-420e-9829-c9584b5afa3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:02:26.082238 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.082204 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-xks-ca" pod "6c10247a-e55b-420e-9829-c9584b5afa3e" (UID: "6c10247a-e55b-420e-9829-c9584b5afa3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:02:26.082390 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.082297 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c10247a-e55b-420e-9829-c9584b5afa3e-kube-api-access-pfv22" pod "6c10247a-e55b-420e-9829-c9584b5afa3e" (UID: "6c10247a-e55b-420e-9829-c9584b5afa3e"). InnerVolumeSpecName "kube-api-access-pfv22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:02:26.181180 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.181091 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfv22\" (UniqueName: \"kubernetes.io/projected/6c10247a-e55b-420e-9829-c9584b5afa3e-kube-api-access-pfv22\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:02:26.181180 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.181121 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:02:26.181180 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.181133 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c10247a-e55b-420e-9829-c9584b5afa3e-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:02:26.849890 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.849807 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" event={"ID":"6c10247a-e55b-420e-9829-c9584b5afa3e","Type":"ContainerDied","Data":"aa4af97456c9a0afd233bf9b3f8fb39248464ab8c33c1c98b5d56bcbcc4de16c"} Nov 01 22:02:26.849890 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.849868 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4af97456c9a0afd233bf9b3f8fb39248464ab8c33c1c98b5d56bcbcc4de16c" Nov 01 22:02:26.849890 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:26.849817 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w" Nov 01 22:02:28.931076 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:28.931034 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:02:35.886797 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:35.886758 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:02:46.209016 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:02:46.208973 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:02:49.254261 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:49.254224 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:02:58.565605 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:02:58.565564 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:00.109715 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:00.109679 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:03:00.362614 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:00.362523 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:03:03.343512 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:03.343465 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:03.344201 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:03.344163 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:03.568527 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:03.568489 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:03.568806 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:03.568784 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:03.754524 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:03.754482 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:03.754836 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:03.754801 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:10.915522 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.915483 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd"] Nov 01 22:03:10.915957 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.915926 5633 cpu_manager.go:414] "containerMap: removing container" podUID="6c10247a-e55b-420e-9829-c9584b5afa3e" containerName="revoke-keys" Nov 01 22:03:10.915957 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.915939 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c10247a-e55b-420e-9829-c9584b5afa3e" containerName="revoke-keys" Nov 01 22:03:10.916063 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.916053 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c10247a-e55b-420e-9829-c9584b5afa3e" containerName="revoke-keys" Nov 01 22:03:10.919284 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.919267 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:10.996550 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.996516 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bee66e-427e-4db4-9112-3256841d368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:10.996734 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.996578 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnww7\" (UniqueName: \"kubernetes.io/projected/37bee66e-427e-4db4-9112-3256841d368c-kube-api-access-mnww7\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:10.996734 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:10.996684 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bee66e-427e-4db4-9112-3256841d368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.021371 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.021310 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:03:11.021371 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.021349 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:03:11.021371 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.021367 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:03:11.021856 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.021838 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:03:11.024528 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.024506 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd"] Nov 01 22:03:11.098146 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.098109 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bee66e-427e-4db4-9112-3256841d368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.098366 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.098169 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnww7\" (UniqueName: \"kubernetes.io/projected/37bee66e-427e-4db4-9112-3256841d368c-kube-api-access-mnww7\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.098366 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.098208 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bee66e-427e-4db4-9112-3256841d368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.098888 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.098857 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bee66e-427e-4db4-9112-3256841d368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.100738 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.100714 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bee66e-427e-4db4-9112-3256841d368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.154031 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.153993 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnww7\" (UniqueName: \"kubernetes.io/projected/37bee66e-427e-4db4-9112-3256841d368c-kube-api-access-mnww7\") pod \"maas-api-revoke-sub-ab8d7e49-4kdcd\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.232722 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.232688 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:11.604176 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:11.604145 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd"] Nov 01 22:03:11.605512 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:03:11.605477 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bee66e_427e_4db4_9112_3256841d368c.slice/crio-92fab731319863e698987c61650040de3df714ff1e6f0c4dcaa08d085f732e4b WatchSource:0}: Error finding container 92fab731319863e698987c61650040de3df714ff1e6f0c4dcaa08d085f732e4b: Status 404 returned error can't find the container with id 92fab731319863e698987c61650040de3df714ff1e6f0c4dcaa08d085f732e4b Nov 01 22:03:12.219773 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:12.219742 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" event={"ID":"37bee66e-427e-4db4-9112-3256841d368c","Type":"ContainerStarted","Data":"956c9b792e643c981fe02fead98cbc47356c6fdc0551fbe8e4a94cf9214a23a9"} Nov 01 22:03:12.220172 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:12.219790 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" event={"ID":"37bee66e-427e-4db4-9112-3256841d368c","Type":"ContainerStarted","Data":"92fab731319863e698987c61650040de3df714ff1e6f0c4dcaa08d085f732e4b"} Nov 01 22:03:12.380127 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:12.380070 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" podStartSLOduration=2.380058682 podStartE2EDuration="2.380058682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:12.379742502 +0000 UTC m=+7200.054001758" watchObservedRunningTime="2026-11-01 22:03:12.380058682 +0000 UTC m=+7200.054317965" Nov 01 22:03:13.228148 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:13.228101 5633 generic.go:356] "Generic (PLEG): container finished" podID="37bee66e-427e-4db4-9112-3256841d368c" containerID="956c9b792e643c981fe02fead98cbc47356c6fdc0551fbe8e4a94cf9214a23a9" exitCode=0 Nov 01 22:03:13.228624 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:13.228185 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" event={"ID":"37bee66e-427e-4db4-9112-3256841d368c","Type":"ContainerDied","Data":"956c9b792e643c981fe02fead98cbc47356c6fdc0551fbe8e4a94cf9214a23a9"} Nov 01 22:03:14.371340 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.371302 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:14.537531 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.537444 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bee66e-427e-4db4-9112-3256841d368c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bee66e-427e-4db4-9112-3256841d368c-maas-api-openshift-ca\") pod \"37bee66e-427e-4db4-9112-3256841d368c\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " Nov 01 22:03:14.537683 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.537575 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bee66e-427e-4db4-9112-3256841d368c-kube-api-access-mnww7\" (UniqueName: \"kubernetes.io/projected/37bee66e-427e-4db4-9112-3256841d368c-kube-api-access-mnww7\") pod \"37bee66e-427e-4db4-9112-3256841d368c\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " Nov 01 22:03:14.537683 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.537641 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37bee66e-427e-4db4-9112-3256841d368c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bee66e-427e-4db4-9112-3256841d368c-maas-api-xks-ca\") pod \"37bee66e-427e-4db4-9112-3256841d368c\" (UID: \"37bee66e-427e-4db4-9112-3256841d368c\") " Nov 01 22:03:14.537885 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.537857 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bee66e-427e-4db4-9112-3256841d368c-maas-api-openshift-ca" pod "37bee66e-427e-4db4-9112-3256841d368c" (UID: "37bee66e-427e-4db4-9112-3256841d368c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:14.539937 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.539905 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37bee66e-427e-4db4-9112-3256841d368c-maas-api-xks-ca" pod "37bee66e-427e-4db4-9112-3256841d368c" (UID: "37bee66e-427e-4db4-9112-3256841d368c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:03:14.540082 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.539920 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bee66e-427e-4db4-9112-3256841d368c-kube-api-access-mnww7" pod "37bee66e-427e-4db4-9112-3256841d368c" (UID: "37bee66e-427e-4db4-9112-3256841d368c"). InnerVolumeSpecName "kube-api-access-mnww7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:14.638436 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.638406 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bee66e-427e-4db4-9112-3256841d368c-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:03:14.638436 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.638434 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bee66e-427e-4db4-9112-3256841d368c-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:03:14.638710 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:14.638449 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnww7\" (UniqueName: \"kubernetes.io/projected/37bee66e-427e-4db4-9112-3256841d368c-kube-api-access-mnww7\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:03:15.240311 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:15.240284 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" Nov 01 22:03:15.240540 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:15.240318 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8d7e49-4kdcd" event={"ID":"37bee66e-427e-4db4-9112-3256841d368c","Type":"ContainerDied","Data":"92fab731319863e698987c61650040de3df714ff1e6f0c4dcaa08d085f732e4b"} Nov 01 22:03:15.240540 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:15.240392 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92fab731319863e698987c61650040de3df714ff1e6f0c4dcaa08d085f732e4b" Nov 01 22:03:16.158304 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:03:16.158268 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:03:21.646800 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.646706 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs"] Nov 01 22:03:21.647298 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.647280 5633 cpu_manager.go:414] "containerMap: removing container" podUID="37bee66e-427e-4db4-9112-3256841d368c" containerName="revoke-keys" Nov 01 22:03:21.647379 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.647300 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bee66e-427e-4db4-9112-3256841d368c" containerName="revoke-keys" Nov 01 22:03:21.647428 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.647398 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bee66e-427e-4db4-9112-3256841d368c" containerName="revoke-keys" Nov 01 22:03:21.650617 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.650598 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.706838 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.706807 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs"] Nov 01 22:03:21.714437 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.714407 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.714437 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.714442 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77nx\" (UniqueName: \"kubernetes.io/projected/ab564d94-16e6-49bc-a2d1-018c50d7cfde-kube-api-access-h77nx\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.714689 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.714547 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.751990 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.751947 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:03:21.751990 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.751950 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:03:21.751990 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.751986 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:03:21.752246 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.752026 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:03:21.815136 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.815104 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.815371 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.815228 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.815371 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.815255 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77nx\" (UniqueName: \"kubernetes.io/projected/ab564d94-16e6-49bc-a2d1-018c50d7cfde-kube-api-access-h77nx\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.815818 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.815796 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.817769 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.817748 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.871512 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.871472 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77nx\" (UniqueName: \"kubernetes.io/projected/ab564d94-16e6-49bc-a2d1-018c50d7cfde-kube-api-access-h77nx\") pod \"maas-api-revoke-sub-0eb8e4bd-mmgvs\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:21.965880 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:21.965846 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:22.339257 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:22.339231 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs"] Nov 01 22:03:22.340607 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:03:22.340580 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab564d94_16e6_49bc_a2d1_018c50d7cfde.slice/crio-47cff070535b3a1b70571cd5443f1d37c2f7639f1f67ba8d28683d385416adb4 WatchSource:0}: Error finding container 47cff070535b3a1b70571cd5443f1d37c2f7639f1f67ba8d28683d385416adb4: Status 404 returned error can't find the container with id 47cff070535b3a1b70571cd5443f1d37c2f7639f1f67ba8d28683d385416adb4 Nov 01 22:03:23.300776 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:23.300738 5633 generic.go:356] "Generic (PLEG): container finished" podID="ab564d94-16e6-49bc-a2d1-018c50d7cfde" containerID="8c91a5b19a534fa0a6287bf3dfa0ce76dd12a844c75b140030451ccf526c8154" exitCode=0 Nov 01 22:03:23.301234 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:23.300829 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" event={"ID":"ab564d94-16e6-49bc-a2d1-018c50d7cfde","Type":"ContainerDied","Data":"8c91a5b19a534fa0a6287bf3dfa0ce76dd12a844c75b140030451ccf526c8154"} Nov 01 22:03:23.301234 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:23.300885 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" event={"ID":"ab564d94-16e6-49bc-a2d1-018c50d7cfde","Type":"ContainerStarted","Data":"47cff070535b3a1b70571cd5443f1d37c2f7639f1f67ba8d28683d385416adb4"} Nov 01 22:03:23.762732 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:23.762688 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:03:24.467028 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.466984 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:24.537654 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.537616 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-xks-ca\") pod \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " Nov 01 22:03:24.537849 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.537770 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab564d94-16e6-49bc-a2d1-018c50d7cfde-kube-api-access-h77nx\" (UniqueName: \"kubernetes.io/projected/ab564d94-16e6-49bc-a2d1-018c50d7cfde-kube-api-access-h77nx\") pod \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " Nov 01 22:03:24.537921 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.537874 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-openshift-ca\") pod \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\" (UID: \"ab564d94-16e6-49bc-a2d1-018c50d7cfde\") " Nov 01 22:03:24.538595 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.538302 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-openshift-ca" pod "ab564d94-16e6-49bc-a2d1-018c50d7cfde" (UID: "ab564d94-16e6-49bc-a2d1-018c50d7cfde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:24.540652 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.540617 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-xks-ca" pod "ab564d94-16e6-49bc-a2d1-018c50d7cfde" (UID: "ab564d94-16e6-49bc-a2d1-018c50d7cfde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:03:24.540963 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.540938 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab564d94-16e6-49bc-a2d1-018c50d7cfde-kube-api-access-h77nx" pod "ab564d94-16e6-49bc-a2d1-018c50d7cfde" (UID: "ab564d94-16e6-49bc-a2d1-018c50d7cfde"). InnerVolumeSpecName "kube-api-access-h77nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:24.639783 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.639677 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h77nx\" (UniqueName: \"kubernetes.io/projected/ab564d94-16e6-49bc-a2d1-018c50d7cfde-kube-api-access-h77nx\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:03:24.639783 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.639727 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:03:24.639783 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:24.639745 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab564d94-16e6-49bc-a2d1-018c50d7cfde-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:03:25.357746 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:25.357649 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" event={"ID":"ab564d94-16e6-49bc-a2d1-018c50d7cfde","Type":"ContainerDied","Data":"47cff070535b3a1b70571cd5443f1d37c2f7639f1f67ba8d28683d385416adb4"} Nov 01 22:03:25.357746 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:25.357715 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47cff070535b3a1b70571cd5443f1d37c2f7639f1f67ba8d28683d385416adb4" Nov 01 22:03:25.358036 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:25.357811 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb8e4bd-mmgvs" Nov 01 22:03:34.486042 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:34.486003 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:03:43.278760 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:43.278722 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:03:46.159929 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:03:46.159885 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:03:58.964692 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:03:58.964593 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:04.894274 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:04.894238 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:04:12.113384 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.113343 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n"] Nov 01 22:04:12.113879 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.113824 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ab564d94-16e6-49bc-a2d1-018c50d7cfde" containerName="revoke-keys" Nov 01 22:04:12.113879 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.113837 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab564d94-16e6-49bc-a2d1-018c50d7cfde" containerName="revoke-keys" Nov 01 22:04:12.113960 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.113939 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab564d94-16e6-49bc-a2d1-018c50d7cfde" containerName="revoke-keys" Nov 01 22:04:12.117180 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.117163 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.169652 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.169611 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:04:12.173075 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.173040 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n"] Nov 01 22:04:12.207532 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.207497 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.207711 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.207535 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2m9h\" (UniqueName: \"kubernetes.io/projected/9835e43d-f139-469b-9eed-1d67e958512b-kube-api-access-s2m9h\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.207711 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.207665 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.217567 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.217531 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:04:12.217567 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.217557 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:04:12.218950 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.218931 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:04:12.309023 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.308983 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.309023 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.309025 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2m9h\" (UniqueName: \"kubernetes.io/projected/9835e43d-f139-469b-9eed-1d67e958512b-kube-api-access-s2m9h\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.309306 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.309114 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.309849 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.309825 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.311648 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.311630 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.365194 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.365118 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2m9h\" (UniqueName: \"kubernetes.io/projected/9835e43d-f139-469b-9eed-1d67e958512b-kube-api-access-s2m9h\") pod \"maas-api-revoke-sub-028c9594-t4r7n\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.431708 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.431672 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:12.804409 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:12.804366 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n"] Nov 01 22:04:12.804852 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:04:12.804828 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9835e43d_f139_469b_9eed_1d67e958512b.slice/crio-e291bede287172b351026c8c7200ab506d66f3c1198b3e300845f584c7ca44e0 WatchSource:0}: Error finding container e291bede287172b351026c8c7200ab506d66f3c1198b3e300845f584c7ca44e0: Status 404 returned error can't find the container with id e291bede287172b351026c8c7200ab506d66f3c1198b3e300845f584c7ca44e0 Nov 01 22:04:13.796631 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:13.796592 5633 generic.go:356] "Generic (PLEG): container finished" podID="9835e43d-f139-469b-9eed-1d67e958512b" containerID="d780ec68999ff55dbfd0a1b76756bc3e578651e2f272c5211f59c2c2090c2679" exitCode=0 Nov 01 22:04:13.797074 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:13.796681 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" event={"ID":"9835e43d-f139-469b-9eed-1d67e958512b","Type":"ContainerDied","Data":"d780ec68999ff55dbfd0a1b76756bc3e578651e2f272c5211f59c2c2090c2679"} Nov 01 22:04:13.797074 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:13.796733 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" event={"ID":"9835e43d-f139-469b-9eed-1d67e958512b","Type":"ContainerStarted","Data":"e291bede287172b351026c8c7200ab506d66f3c1198b3e300845f584c7ca44e0"} Nov 01 22:04:14.944186 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:14.944157 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:15.037985 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.037946 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-xks-ca\") pod \"9835e43d-f139-469b-9eed-1d67e958512b\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " Nov 01 22:04:15.038187 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.038057 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-openshift-ca\") pod \"9835e43d-f139-469b-9eed-1d67e958512b\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " Nov 01 22:04:15.038187 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.038116 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9835e43d-f139-469b-9eed-1d67e958512b-kube-api-access-s2m9h\" (UniqueName: \"kubernetes.io/projected/9835e43d-f139-469b-9eed-1d67e958512b-kube-api-access-s2m9h\") pod \"9835e43d-f139-469b-9eed-1d67e958512b\" (UID: \"9835e43d-f139-469b-9eed-1d67e958512b\") " Nov 01 22:04:15.038448 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.038424 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-openshift-ca" pod "9835e43d-f139-469b-9eed-1d67e958512b" (UID: "9835e43d-f139-469b-9eed-1d67e958512b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:15.040470 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.040435 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-xks-ca" pod "9835e43d-f139-469b-9eed-1d67e958512b" (UID: "9835e43d-f139-469b-9eed-1d67e958512b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:04:15.040601 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.040540 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9835e43d-f139-469b-9eed-1d67e958512b-kube-api-access-s2m9h" pod "9835e43d-f139-469b-9eed-1d67e958512b" (UID: "9835e43d-f139-469b-9eed-1d67e958512b"). InnerVolumeSpecName "kube-api-access-s2m9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:15.139782 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.139690 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:15.139782 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.139725 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2m9h\" (UniqueName: \"kubernetes.io/projected/9835e43d-f139-469b-9eed-1d67e958512b-kube-api-access-s2m9h\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:15.139782 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.139735 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9835e43d-f139-469b-9eed-1d67e958512b-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:15.807995 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.807959 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" Nov 01 22:04:15.807995 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.807990 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-028c9594-t4r7n" event={"ID":"9835e43d-f139-469b-9eed-1d67e958512b","Type":"ContainerDied","Data":"e291bede287172b351026c8c7200ab506d66f3c1198b3e300845f584c7ca44e0"} Nov 01 22:04:15.808223 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:15.808043 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e291bede287172b351026c8c7200ab506d66f3c1198b3e300845f584c7ca44e0" Nov 01 22:04:16.156157 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:04:16.156071 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:04:23.319374 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.319314 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm"] Nov 01 22:04:23.319840 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.319825 5633 cpu_manager.go:414] "containerMap: removing container" podUID="9835e43d-f139-469b-9eed-1d67e958512b" containerName="revoke-keys" Nov 01 22:04:23.319896 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.319842 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9835e43d-f139-469b-9eed-1d67e958512b" containerName="revoke-keys" Nov 01 22:04:23.319945 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.319935 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9835e43d-f139-469b-9eed-1d67e958512b" containerName="revoke-keys" Nov 01 22:04:23.323602 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.323574 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.379364 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.379293 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm"] Nov 01 22:04:23.423824 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.423782 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:04:23.423824 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.423798 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:04:23.423824 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.423805 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:04:23.425032 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.425014 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:04:23.433412 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.433388 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.433546 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.433523 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.433700 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.433676 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlgwf\" (UniqueName: \"kubernetes.io/projected/3df73ce0-9e69-4988-bb01-7cfc33850ba1-kube-api-access-xlgwf\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.535279 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.535237 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.535516 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.535305 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.535516 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.535430 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlgwf\" (UniqueName: \"kubernetes.io/projected/3df73ce0-9e69-4988-bb01-7cfc33850ba1-kube-api-access-xlgwf\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.535988 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.535965 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.537959 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.537937 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.591709 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.591626 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlgwf\" (UniqueName: \"kubernetes.io/projected/3df73ce0-9e69-4988-bb01-7cfc33850ba1-kube-api-access-xlgwf\") pod \"maas-api-revoke-sub-54a5177f-spfwm\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:23.636673 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:23.636637 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:24.011813 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:24.011782 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm"] Nov 01 22:04:24.012729 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:04:24.012703 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df73ce0_9e69_4988_bb01_7cfc33850ba1.slice/crio-4e2b5b7affadcb1740c818f52912868936cc185e30cf4a11565e58755f368dcd WatchSource:0}: Error finding container 4e2b5b7affadcb1740c818f52912868936cc185e30cf4a11565e58755f368dcd: Status 404 returned error can't find the container with id 4e2b5b7affadcb1740c818f52912868936cc185e30cf4a11565e58755f368dcd Nov 01 22:04:24.861362 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:24.861311 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" event={"ID":"3df73ce0-9e69-4988-bb01-7cfc33850ba1","Type":"ContainerStarted","Data":"3770fa3513e56df61a9314dbf3296d232992fea221b711478c95abed2ab161b0"} Nov 01 22:04:24.861854 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:24.861376 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" event={"ID":"3df73ce0-9e69-4988-bb01-7cfc33850ba1","Type":"ContainerStarted","Data":"4e2b5b7affadcb1740c818f52912868936cc185e30cf4a11565e58755f368dcd"} Nov 01 22:04:24.975129 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:24.975073 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" podStartSLOduration=1.975057265 podStartE2EDuration="1.975057265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:24.972412846 +0000 UTC m=+7272.646672092" watchObservedRunningTime="2026-11-01 22:04:24.975057265 +0000 UTC m=+7272.649316520" Nov 01 22:04:25.868633 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:25.868595 5633 generic.go:356] "Generic (PLEG): container finished" podID="3df73ce0-9e69-4988-bb01-7cfc33850ba1" containerID="3770fa3513e56df61a9314dbf3296d232992fea221b711478c95abed2ab161b0" exitCode=0 Nov 01 22:04:25.869059 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:25.868650 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" event={"ID":"3df73ce0-9e69-4988-bb01-7cfc33850ba1","Type":"ContainerDied","Data":"3770fa3513e56df61a9314dbf3296d232992fea221b711478c95abed2ab161b0"} Nov 01 22:04:26.757983 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:26.757944 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:04:27.045719 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.045692 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:27.174369 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.174183 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-openshift-ca\") pod \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " Nov 01 22:04:27.174369 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.174233 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-xks-ca\") pod \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " Nov 01 22:04:27.178616 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.174673 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df73ce0-9e69-4988-bb01-7cfc33850ba1-kube-api-access-xlgwf\" (UniqueName: \"kubernetes.io/projected/3df73ce0-9e69-4988-bb01-7cfc33850ba1-kube-api-access-xlgwf\") pod \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\" (UID: \"3df73ce0-9e69-4988-bb01-7cfc33850ba1\") " Nov 01 22:04:27.178616 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.174692 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-openshift-ca" pod "3df73ce0-9e69-4988-bb01-7cfc33850ba1" (UID: "3df73ce0-9e69-4988-bb01-7cfc33850ba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:27.178616 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.178560 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df73ce0-9e69-4988-bb01-7cfc33850ba1-kube-api-access-xlgwf" pod "3df73ce0-9e69-4988-bb01-7cfc33850ba1" (UID: "3df73ce0-9e69-4988-bb01-7cfc33850ba1"). InnerVolumeSpecName "kube-api-access-xlgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:27.179356 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.179295 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlgwf\" (UniqueName: \"kubernetes.io/projected/3df73ce0-9e69-4988-bb01-7cfc33850ba1-kube-api-access-xlgwf\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:27.179505 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.179352 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:27.186523 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.186486 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-xks-ca" pod "3df73ce0-9e69-4988-bb01-7cfc33850ba1" (UID: "3df73ce0-9e69-4988-bb01-7cfc33850ba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:04:27.280629 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.280527 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df73ce0-9e69-4988-bb01-7cfc33850ba1-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:27.883464 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.883426 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" Nov 01 22:04:27.883691 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.883434 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5177f-spfwm" event={"ID":"3df73ce0-9e69-4988-bb01-7cfc33850ba1","Type":"ContainerDied","Data":"4e2b5b7affadcb1740c818f52912868936cc185e30cf4a11565e58755f368dcd"} Nov 01 22:04:27.883691 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:27.883557 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e2b5b7affadcb1740c818f52912868936cc185e30cf4a11565e58755f368dcd" Nov 01 22:04:33.992956 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:33.992917 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt"] Nov 01 22:04:33.993424 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:33.993412 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3df73ce0-9e69-4988-bb01-7cfc33850ba1" containerName="revoke-keys" Nov 01 22:04:33.993472 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:33.993424 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df73ce0-9e69-4988-bb01-7cfc33850ba1" containerName="revoke-keys" Nov 01 22:04:33.993539 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:33.993529 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df73ce0-9e69-4988-bb01-7cfc33850ba1" containerName="revoke-keys" Nov 01 22:04:33.996980 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:33.996956 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.051945 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.051914 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt"] Nov 01 22:04:34.096770 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.096733 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:04:34.096770 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.096748 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:04:34.096770 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.096769 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:04:34.096770 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.096744 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.097111 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.096825 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjkc\" (UniqueName: \"kubernetes.io/projected/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-kube-api-access-xhjkc\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.097111 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.096876 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.098647 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.098619 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:04:34.197533 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.197498 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.197533 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.197542 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjkc\" (UniqueName: \"kubernetes.io/projected/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-kube-api-access-xhjkc\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.197829 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.197575 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.198294 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.198273 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.200145 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.200113 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.253726 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.253648 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjkc\" (UniqueName: \"kubernetes.io/projected/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-kube-api-access-xhjkc\") pod \"maas-api-revoke-sub-f201f7fe-nwswt\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.312378 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.312310 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:34.686058 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.686032 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt"] Nov 01 22:04:34.687437 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:04:34.687395 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb92700_3f1c_46f1_a2f2_67f9fa501d8b.slice/crio-0a818c94da7cb1b1fd134d539e52d0206e2e4755909708cf01a93c7f3ca8b192 WatchSource:0}: Error finding container 0a818c94da7cb1b1fd134d539e52d0206e2e4755909708cf01a93c7f3ca8b192: Status 404 returned error can't find the container with id 0a818c94da7cb1b1fd134d539e52d0206e2e4755909708cf01a93c7f3ca8b192 Nov 01 22:04:34.974848 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:34.974807 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" event={"ID":"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b","Type":"ContainerStarted","Data":"0a818c94da7cb1b1fd134d539e52d0206e2e4755909708cf01a93c7f3ca8b192"} Nov 01 22:04:35.982234 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:35.982203 5633 generic.go:356] "Generic (PLEG): container finished" podID="fcb92700-3f1c-46f1-a2f2-67f9fa501d8b" containerID="3b13210de70275efe9234ce55674ac53dda028d3656fc3bbd82ee0d8968fda70" exitCode=0 Nov 01 22:04:35.982749 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:35.982294 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" event={"ID":"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b","Type":"ContainerDied","Data":"3b13210de70275efe9234ce55674ac53dda028d3656fc3bbd82ee0d8968fda70"} Nov 01 22:04:37.134412 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.134383 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:37.222392 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.222323 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-xks-ca\") pod \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " Nov 01 22:04:37.222577 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.222410 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-kube-api-access-xhjkc\" (UniqueName: \"kubernetes.io/projected/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-kube-api-access-xhjkc\") pod \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " Nov 01 22:04:37.222577 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.222436 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-openshift-ca\") pod \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\" (UID: \"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b\") " Nov 01 22:04:37.222877 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.222853 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-openshift-ca" pod "fcb92700-3f1c-46f1-a2f2-67f9fa501d8b" (UID: "fcb92700-3f1c-46f1-a2f2-67f9fa501d8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:37.224675 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.224640 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-xks-ca" pod "fcb92700-3f1c-46f1-a2f2-67f9fa501d8b" (UID: "fcb92700-3f1c-46f1-a2f2-67f9fa501d8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:04:37.224809 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.224701 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-kube-api-access-xhjkc" pod "fcb92700-3f1c-46f1-a2f2-67f9fa501d8b" (UID: "fcb92700-3f1c-46f1-a2f2-67f9fa501d8b"). InnerVolumeSpecName "kube-api-access-xhjkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:37.323794 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.323697 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjkc\" (UniqueName: \"kubernetes.io/projected/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-kube-api-access-xhjkc\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:37.323794 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.323735 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:37.323794 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.323745 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb92700-3f1c-46f1-a2f2-67f9fa501d8b-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:37.995295 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.995263 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" Nov 01 22:04:37.995518 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.995259 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f201f7fe-nwswt" event={"ID":"fcb92700-3f1c-46f1-a2f2-67f9fa501d8b","Type":"ContainerDied","Data":"0a818c94da7cb1b1fd134d539e52d0206e2e4755909708cf01a93c7f3ca8b192"} Nov 01 22:04:37.995518 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:37.995382 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a818c94da7cb1b1fd134d539e52d0206e2e4755909708cf01a93c7f3ca8b192" Nov 01 22:04:40.788905 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:40.788869 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:41.019910 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:41.019873 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:45.154929 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.154890 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc"] Nov 01 22:04:45.155433 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.155369 5633 cpu_manager.go:414] "containerMap: removing container" podUID="fcb92700-3f1c-46f1-a2f2-67f9fa501d8b" containerName="revoke-keys" Nov 01 22:04:45.155433 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.155381 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb92700-3f1c-46f1-a2f2-67f9fa501d8b" containerName="revoke-keys" Nov 01 22:04:45.155525 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.155467 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb92700-3f1c-46f1-a2f2-67f9fa501d8b" containerName="revoke-keys" Nov 01 22:04:45.158858 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.158837 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.205419 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.205378 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29383b31-0598-4821-8671-89805998a3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.205595 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.205504 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29383b31-0598-4821-8671-89805998a3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.205595 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.205553 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lkv\" (UniqueName: \"kubernetes.io/projected/29383b31-0598-4821-8671-89805998a3ee-kube-api-access-d9lkv\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.214229 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.214195 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc"] Nov 01 22:04:45.259501 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.259469 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:04:45.259501 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.259477 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:04:45.259718 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.259515 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:04:45.260183 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.260159 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:04:45.306784 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.306754 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29383b31-0598-4821-8671-89805998a3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.306998 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.306820 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lkv\" (UniqueName: \"kubernetes.io/projected/29383b31-0598-4821-8671-89805998a3ee-kube-api-access-d9lkv\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.306998 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.306902 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29383b31-0598-4821-8671-89805998a3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.307631 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.307605 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29383b31-0598-4821-8671-89805998a3ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.309393 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.309370 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29383b31-0598-4821-8671-89805998a3ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.363811 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.363769 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lkv\" (UniqueName: \"kubernetes.io/projected/29383b31-0598-4821-8671-89805998a3ee-kube-api-access-d9lkv\") pod \"maas-api-revoke-sub-2afc9869-sffhc\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.474092 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.474056 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:45.847463 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:45.847431 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc"] Nov 01 22:04:45.848195 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:04:45.848173 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29383b31_0598_4821_8671_89805998a3ee.slice/crio-6ff5a32cc9fa7468315253df1f96b57f888e00184d94c856793c2c94f80899b3 WatchSource:0}: Error finding container 6ff5a32cc9fa7468315253df1f96b57f888e00184d94c856793c2c94f80899b3: Status 404 returned error can't find the container with id 6ff5a32cc9fa7468315253df1f96b57f888e00184d94c856793c2c94f80899b3 Nov 01 22:04:46.046358 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:46.046298 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" event={"ID":"29383b31-0598-4821-8671-89805998a3ee","Type":"ContainerStarted","Data":"6ff5a32cc9fa7468315253df1f96b57f888e00184d94c856793c2c94f80899b3"} Nov 01 22:04:46.156656 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:04:46.156613 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:04:47.054153 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:47.054120 5633 generic.go:356] "Generic (PLEG): container finished" podID="29383b31-0598-4821-8671-89805998a3ee" containerID="d45dd3d2bb9583f196dfa362be3e3b7a9b2b2824238421d1d3a3b3a818d31a16" exitCode=0 Nov 01 22:04:47.054361 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:47.054216 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" event={"ID":"29383b31-0598-4821-8671-89805998a3ee","Type":"ContainerDied","Data":"d45dd3d2bb9583f196dfa362be3e3b7a9b2b2824238421d1d3a3b3a818d31a16"} Nov 01 22:04:48.196238 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.196210 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:48.241971 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.241936 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29383b31-0598-4821-8671-89805998a3ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29383b31-0598-4821-8671-89805998a3ee-maas-api-openshift-ca\") pod \"29383b31-0598-4821-8671-89805998a3ee\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " Nov 01 22:04:48.241971 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.241974 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29383b31-0598-4821-8671-89805998a3ee-kube-api-access-d9lkv\" (UniqueName: \"kubernetes.io/projected/29383b31-0598-4821-8671-89805998a3ee-kube-api-access-d9lkv\") pod \"29383b31-0598-4821-8671-89805998a3ee\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " Nov 01 22:04:48.242192 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.242105 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29383b31-0598-4821-8671-89805998a3ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29383b31-0598-4821-8671-89805998a3ee-maas-api-xks-ca\") pod \"29383b31-0598-4821-8671-89805998a3ee\" (UID: \"29383b31-0598-4821-8671-89805998a3ee\") " Nov 01 22:04:48.242361 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.242305 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29383b31-0598-4821-8671-89805998a3ee-maas-api-openshift-ca" pod "29383b31-0598-4821-8671-89805998a3ee" (UID: "29383b31-0598-4821-8671-89805998a3ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:48.244412 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.244372 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29383b31-0598-4821-8671-89805998a3ee-maas-api-xks-ca" pod "29383b31-0598-4821-8671-89805998a3ee" (UID: "29383b31-0598-4821-8671-89805998a3ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:04:48.244541 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.244464 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29383b31-0598-4821-8671-89805998a3ee-kube-api-access-d9lkv" pod "29383b31-0598-4821-8671-89805998a3ee" (UID: "29383b31-0598-4821-8671-89805998a3ee"). InnerVolumeSpecName "kube-api-access-d9lkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:48.343855 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.343772 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29383b31-0598-4821-8671-89805998a3ee-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:48.343855 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.343805 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29383b31-0598-4821-8671-89805998a3ee-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:48.343855 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:48.343816 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9lkv\" (UniqueName: \"kubernetes.io/projected/29383b31-0598-4821-8671-89805998a3ee-kube-api-access-d9lkv\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:49.070184 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:49.070136 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" event={"ID":"29383b31-0598-4821-8671-89805998a3ee","Type":"ContainerDied","Data":"6ff5a32cc9fa7468315253df1f96b57f888e00184d94c856793c2c94f80899b3"} Nov 01 22:04:49.070439 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:49.070155 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afc9869-sffhc" Nov 01 22:04:49.070439 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:49.070239 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ff5a32cc9fa7468315253df1f96b57f888e00184d94c856793c2c94f80899b3" Nov 01 22:04:55.834207 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.834163 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq"] Nov 01 22:04:55.836630 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.834694 5633 cpu_manager.go:414] "containerMap: removing container" podUID="29383b31-0598-4821-8671-89805998a3ee" containerName="revoke-keys" Nov 01 22:04:55.836630 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.834708 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29383b31-0598-4821-8671-89805998a3ee" containerName="revoke-keys" Nov 01 22:04:55.836630 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.834782 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="29383b31-0598-4821-8671-89805998a3ee" containerName="revoke-keys" Nov 01 22:04:55.838014 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.837992 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:55.893383 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.893322 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq"] Nov 01 22:04:55.938035 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.938000 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:04:55.938219 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.938044 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:04:55.939463 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.939439 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:04:55.939463 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.939445 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:04:55.972744 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.972706 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:55.972950 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.972840 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:55.972950 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:55.972865 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qbx\" (UniqueName: \"kubernetes.io/projected/4c1f2d12-8aca-4688-bbbf-3925080d041c-kube-api-access-c2qbx\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.073822 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.073779 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.073822 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.073823 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qbx\" (UniqueName: \"kubernetes.io/projected/4c1f2d12-8aca-4688-bbbf-3925080d041c-kube-api-access-c2qbx\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.074086 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.073927 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.074587 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.074557 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.076531 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.076498 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.130038 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.129965 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qbx\" (UniqueName: \"kubernetes.io/projected/4c1f2d12-8aca-4688-bbbf-3925080d041c-kube-api-access-c2qbx\") pod \"maas-api-revoke-sub-0688a8f2-j8qgq\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.153859 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.153821 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:56.531178 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:56.531145 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq"] Nov 01 22:04:56.531952 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:04:56.531926 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1f2d12_8aca_4688_bbbf_3925080d041c.slice/crio-1172a7f01b612ac9116925776888f1db31edfb3a5a5fce64ab2d33742663f550 WatchSource:0}: Error finding container 1172a7f01b612ac9116925776888f1db31edfb3a5a5fce64ab2d33742663f550: Status 404 returned error can't find the container with id 1172a7f01b612ac9116925776888f1db31edfb3a5a5fce64ab2d33742663f550 Nov 01 22:04:57.134083 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:57.133988 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" event={"ID":"4c1f2d12-8aca-4688-bbbf-3925080d041c","Type":"ContainerStarted","Data":"01d349c4bdefe6caa7b66a208ee81e88b0e3fc6a35db94ad2929e41b073f0216"} Nov 01 22:04:57.134083 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:57.134053 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" event={"ID":"4c1f2d12-8aca-4688-bbbf-3925080d041c","Type":"ContainerStarted","Data":"1172a7f01b612ac9116925776888f1db31edfb3a5a5fce64ab2d33742663f550"} Nov 01 22:04:58.141417 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:58.141382 5633 generic.go:356] "Generic (PLEG): container finished" podID="4c1f2d12-8aca-4688-bbbf-3925080d041c" containerID="01d349c4bdefe6caa7b66a208ee81e88b0e3fc6a35db94ad2929e41b073f0216" exitCode=0 Nov 01 22:04:58.141841 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:58.141469 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" event={"ID":"4c1f2d12-8aca-4688-bbbf-3925080d041c","Type":"ContainerDied","Data":"01d349c4bdefe6caa7b66a208ee81e88b0e3fc6a35db94ad2929e41b073f0216"} Nov 01 22:04:59.255712 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.255675 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:04:59.295551 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.295522 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:04:59.411059 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.410971 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1f2d12-8aca-4688-bbbf-3925080d041c-kube-api-access-c2qbx\" (UniqueName: \"kubernetes.io/projected/4c1f2d12-8aca-4688-bbbf-3925080d041c-kube-api-access-c2qbx\") pod \"4c1f2d12-8aca-4688-bbbf-3925080d041c\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " Nov 01 22:04:59.411266 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.411065 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-xks-ca\") pod \"4c1f2d12-8aca-4688-bbbf-3925080d041c\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " Nov 01 22:04:59.411266 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.411235 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-openshift-ca\") pod \"4c1f2d12-8aca-4688-bbbf-3925080d041c\" (UID: \"4c1f2d12-8aca-4688-bbbf-3925080d041c\") " Nov 01 22:04:59.411838 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.411809 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-openshift-ca" pod "4c1f2d12-8aca-4688-bbbf-3925080d041c" (UID: "4c1f2d12-8aca-4688-bbbf-3925080d041c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:59.413957 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.413923 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-xks-ca" pod "4c1f2d12-8aca-4688-bbbf-3925080d041c" (UID: "4c1f2d12-8aca-4688-bbbf-3925080d041c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:04:59.413957 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.413945 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1f2d12-8aca-4688-bbbf-3925080d041c-kube-api-access-c2qbx" pod "4c1f2d12-8aca-4688-bbbf-3925080d041c" (UID: "4c1f2d12-8aca-4688-bbbf-3925080d041c"). InnerVolumeSpecName "kube-api-access-c2qbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:59.512190 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.512155 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:59.512190 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.512184 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qbx\" (UniqueName: \"kubernetes.io/projected/4c1f2d12-8aca-4688-bbbf-3925080d041c-kube-api-access-c2qbx\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:04:59.512190 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:04:59.512194 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1f2d12-8aca-4688-bbbf-3925080d041c-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:00.154651 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:00.154612 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" Nov 01 22:05:00.154651 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:00.154621 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0688a8f2-j8qgq" event={"ID":"4c1f2d12-8aca-4688-bbbf-3925080d041c","Type":"ContainerDied","Data":"1172a7f01b612ac9116925776888f1db31edfb3a5a5fce64ab2d33742663f550"} Nov 01 22:05:00.154880 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:00.154672 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1172a7f01b612ac9116925776888f1db31edfb3a5a5fce64ab2d33742663f550" Nov 01 22:05:06.608255 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.608211 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh"] Nov 01 22:05:06.609000 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.608978 5633 cpu_manager.go:414] "containerMap: removing container" podUID="4c1f2d12-8aca-4688-bbbf-3925080d041c" containerName="revoke-keys" Nov 01 22:05:06.609048 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.609001 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1f2d12-8aca-4688-bbbf-3925080d041c" containerName="revoke-keys" Nov 01 22:05:06.609179 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.609167 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1f2d12-8aca-4688-bbbf-3925080d041c" containerName="revoke-keys" Nov 01 22:05:06.612632 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.612612 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.667142 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.667108 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh"] Nov 01 22:05:06.684291 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.684251 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h572n\" (UniqueName: \"kubernetes.io/projected/79533546-45d3-4a69-9fb1-a37450126c75-kube-api-access-h572n\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.684498 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.684410 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.684498 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.684473 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.710961 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.710920 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:06.710961 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.710959 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:05:06.711176 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.710964 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:06.713142 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.713125 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:06.785516 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.785479 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h572n\" (UniqueName: \"kubernetes.io/projected/79533546-45d3-4a69-9fb1-a37450126c75-kube-api-access-h572n\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.785688 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.785535 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.785688 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.785577 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.786194 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.786163 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.788192 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.788166 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.841832 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.841794 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h572n\" (UniqueName: \"kubernetes.io/projected/79533546-45d3-4a69-9fb1-a37450126c75-kube-api-access-h572n\") pod \"maas-api-revoke-sub-ccc1ffd1-nxgdh\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:06.926070 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:06.925990 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:07.304992 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:07.304963 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh"] Nov 01 22:05:07.305964 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:05:07.305939 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79533546_45d3_4a69_9fb1_a37450126c75.slice/crio-6e3ff6d5bb6c3e21c641a8704a82748ac04ceba87009939eccc48b4a93ef63d3 WatchSource:0}: Error finding container 6e3ff6d5bb6c3e21c641a8704a82748ac04ceba87009939eccc48b4a93ef63d3: Status 404 returned error can't find the container with id 6e3ff6d5bb6c3e21c641a8704a82748ac04ceba87009939eccc48b4a93ef63d3 Nov 01 22:05:08.202917 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:08.202882 5633 generic.go:356] "Generic (PLEG): container finished" podID="79533546-45d3-4a69-9fb1-a37450126c75" containerID="654f7db20360585ddc498bc1c031d3f15ce56294d2bfd3bb0837311c3f650320" exitCode=0 Nov 01 22:05:08.203379 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:08.202972 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" event={"ID":"79533546-45d3-4a69-9fb1-a37450126c75","Type":"ContainerDied","Data":"654f7db20360585ddc498bc1c031d3f15ce56294d2bfd3bb0837311c3f650320"} Nov 01 22:05:08.203379 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:08.203078 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" event={"ID":"79533546-45d3-4a69-9fb1-a37450126c75","Type":"ContainerStarted","Data":"6e3ff6d5bb6c3e21c641a8704a82748ac04ceba87009939eccc48b4a93ef63d3"} Nov 01 22:05:09.356093 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.356067 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:09.412897 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.412861 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79533546-45d3-4a69-9fb1-a37450126c75-kube-api-access-h572n\" (UniqueName: \"kubernetes.io/projected/79533546-45d3-4a69-9fb1-a37450126c75-kube-api-access-h572n\") pod \"79533546-45d3-4a69-9fb1-a37450126c75\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " Nov 01 22:05:09.413076 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.412977 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-xks-ca\") pod \"79533546-45d3-4a69-9fb1-a37450126c75\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " Nov 01 22:05:09.413076 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.413040 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-openshift-ca\") pod \"79533546-45d3-4a69-9fb1-a37450126c75\" (UID: \"79533546-45d3-4a69-9fb1-a37450126c75\") " Nov 01 22:05:09.413489 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.413459 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-openshift-ca" pod "79533546-45d3-4a69-9fb1-a37450126c75" (UID: "79533546-45d3-4a69-9fb1-a37450126c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:09.415298 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.415267 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-xks-ca" pod "79533546-45d3-4a69-9fb1-a37450126c75" (UID: "79533546-45d3-4a69-9fb1-a37450126c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:09.415423 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.415345 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79533546-45d3-4a69-9fb1-a37450126c75-kube-api-access-h572n" pod "79533546-45d3-4a69-9fb1-a37450126c75" (UID: "79533546-45d3-4a69-9fb1-a37450126c75"). InnerVolumeSpecName "kube-api-access-h572n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:09.514374 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.514304 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h572n\" (UniqueName: \"kubernetes.io/projected/79533546-45d3-4a69-9fb1-a37450126c75-kube-api-access-h572n\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:09.514374 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.514370 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:09.514374 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:09.514380 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79533546-45d3-4a69-9fb1-a37450126c75-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:10.215888 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:10.215853 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" Nov 01 22:05:10.216071 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:10.215852 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc1ffd1-nxgdh" event={"ID":"79533546-45d3-4a69-9fb1-a37450126c75","Type":"ContainerDied","Data":"6e3ff6d5bb6c3e21c641a8704a82748ac04ceba87009939eccc48b4a93ef63d3"} Nov 01 22:05:10.216071 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:10.215972 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3ff6d5bb6c3e21c641a8704a82748ac04ceba87009939eccc48b4a93ef63d3" Nov 01 22:05:16.157168 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:05:16.157128 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:05:17.282029 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.281999 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw"] Nov 01 22:05:17.282527 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.282493 5633 cpu_manager.go:414] "containerMap: removing container" podUID="79533546-45d3-4a69-9fb1-a37450126c75" containerName="revoke-keys" Nov 01 22:05:17.282527 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.282506 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79533546-45d3-4a69-9fb1-a37450126c75" containerName="revoke-keys" Nov 01 22:05:17.282610 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.282572 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="79533546-45d3-4a69-9fb1-a37450126c75" containerName="revoke-keys" Nov 01 22:05:17.286106 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.286088 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.299954 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.299921 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgrgr\" (UniqueName: \"kubernetes.io/projected/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-kube-api-access-pgrgr\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.299954 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.299961 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.300173 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.300037 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.388404 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.388357 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:17.388404 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.388374 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:17.388674 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.388422 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:17.388674 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.388444 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:05:17.391994 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.391970 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw"] Nov 01 22:05:17.401207 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.401171 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.401425 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.401364 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgrgr\" (UniqueName: \"kubernetes.io/projected/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-kube-api-access-pgrgr\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.401425 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.401397 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.401887 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.401866 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.403983 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.403960 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.457770 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.457738 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgrgr\" (UniqueName: \"kubernetes.io/projected/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-kube-api-access-pgrgr\") pod \"maas-api-revoke-sub-9e9f5dbb-8j8sw\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.600406 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.600279 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:17.974825 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:17.974791 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw"] Nov 01 22:05:17.975784 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:05:17.975758 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc539f5a6_0705_4f51_b1a5_7179c92aa7e4.slice/crio-f5cebd2ab57fd1c9baf2d53120865df18e3dc32db7bef023d81cf6491704d203 WatchSource:0}: Error finding container f5cebd2ab57fd1c9baf2d53120865df18e3dc32db7bef023d81cf6491704d203: Status 404 returned error can't find the container with id f5cebd2ab57fd1c9baf2d53120865df18e3dc32db7bef023d81cf6491704d203 Nov 01 22:05:18.289147 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:18.289112 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" event={"ID":"c539f5a6-0705-4f51-b1a5-7179c92aa7e4","Type":"ContainerStarted","Data":"f5cebd2ab57fd1c9baf2d53120865df18e3dc32db7bef023d81cf6491704d203"} Nov 01 22:05:19.296182 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:19.296147 5633 generic.go:356] "Generic (PLEG): container finished" podID="c539f5a6-0705-4f51-b1a5-7179c92aa7e4" containerID="33ab1de02a6418111ca9a5bc75788db7a1e38cf124aa6d08030f9ef7cd138b21" exitCode=0 Nov 01 22:05:19.296661 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:19.296229 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" event={"ID":"c539f5a6-0705-4f51-b1a5-7179c92aa7e4","Type":"ContainerDied","Data":"33ab1de02a6418111ca9a5bc75788db7a1e38cf124aa6d08030f9ef7cd138b21"} Nov 01 22:05:20.450434 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.450409 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:20.532186 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.532138 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-openshift-ca\") pod \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " Nov 01 22:05:20.532425 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.532267 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-kube-api-access-pgrgr\" (UniqueName: \"kubernetes.io/projected/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-kube-api-access-pgrgr\") pod \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " Nov 01 22:05:20.532425 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.532407 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-xks-ca\") pod \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\" (UID: \"c539f5a6-0705-4f51-b1a5-7179c92aa7e4\") " Nov 01 22:05:20.532561 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.532534 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-openshift-ca" pod "c539f5a6-0705-4f51-b1a5-7179c92aa7e4" (UID: "c539f5a6-0705-4f51-b1a5-7179c92aa7e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:20.532825 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.532805 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:20.534640 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.534611 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-kube-api-access-pgrgr" pod "c539f5a6-0705-4f51-b1a5-7179c92aa7e4" (UID: "c539f5a6-0705-4f51-b1a5-7179c92aa7e4"). InnerVolumeSpecName "kube-api-access-pgrgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:20.534779 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.534710 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-xks-ca" pod "c539f5a6-0705-4f51-b1a5-7179c92aa7e4" (UID: "c539f5a6-0705-4f51-b1a5-7179c92aa7e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:20.634363 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.634247 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgrgr\" (UniqueName: \"kubernetes.io/projected/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-kube-api-access-pgrgr\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:20.634363 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:20.634279 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c539f5a6-0705-4f51-b1a5-7179c92aa7e4-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:21.310642 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:21.310602 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" event={"ID":"c539f5a6-0705-4f51-b1a5-7179c92aa7e4","Type":"ContainerDied","Data":"f5cebd2ab57fd1c9baf2d53120865df18e3dc32db7bef023d81cf6491704d203"} Nov 01 22:05:21.310834 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:21.310652 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e9f5dbb-8j8sw" Nov 01 22:05:21.310834 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:21.310654 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5cebd2ab57fd1c9baf2d53120865df18e3dc32db7bef023d81cf6491704d203" Nov 01 22:05:28.019322 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.019285 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq"] Nov 01 22:05:28.019810 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.019795 5633 cpu_manager.go:414] "containerMap: removing container" podUID="c539f5a6-0705-4f51-b1a5-7179c92aa7e4" containerName="revoke-keys" Nov 01 22:05:28.019858 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.019812 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c539f5a6-0705-4f51-b1a5-7179c92aa7e4" containerName="revoke-keys" Nov 01 22:05:28.019926 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.019917 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c539f5a6-0705-4f51-b1a5-7179c92aa7e4" containerName="revoke-keys" Nov 01 22:05:28.023510 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.023493 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.078359 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.078307 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq"] Nov 01 22:05:28.112893 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.112856 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzff\" (UniqueName: \"kubernetes.io/projected/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-kube-api-access-qdzff\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.113080 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.112910 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.113080 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.113021 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.123188 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.123166 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:28.123302 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.123255 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:28.123372 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.123318 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:28.124768 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.124743 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:05:28.214603 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.214562 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzff\" (UniqueName: \"kubernetes.io/projected/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-kube-api-access-qdzff\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.214796 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.214625 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.214796 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.214670 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.215476 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.215455 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.217374 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.217346 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.270803 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.270722 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzff\" (UniqueName: \"kubernetes.io/projected/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-kube-api-access-qdzff\") pod \"maas-api-revoke-sub-a7642b99-nrhzq\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.338143 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.338104 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:28.712649 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:28.712620 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq"] Nov 01 22:05:28.713792 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:05:28.713767 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfd804c_d0e8_4371_98dd_2d17f2a04f77.slice/crio-a279511afe3e69878eadcc9737eb22a128936819a97ce29768c3e2e7a4a16224 WatchSource:0}: Error finding container a279511afe3e69878eadcc9737eb22a128936819a97ce29768c3e2e7a4a16224: Status 404 returned error can't find the container with id a279511afe3e69878eadcc9737eb22a128936819a97ce29768c3e2e7a4a16224 Nov 01 22:05:29.361159 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:29.361056 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" event={"ID":"7cfd804c-d0e8-4371-98dd-2d17f2a04f77","Type":"ContainerStarted","Data":"93a866cce35d2a790d294acf9a7fe3f484b70a3926cbe6316519d9ebfffae4b3"} Nov 01 22:05:29.361159 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:29.361127 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" event={"ID":"7cfd804c-d0e8-4371-98dd-2d17f2a04f77","Type":"ContainerStarted","Data":"a279511afe3e69878eadcc9737eb22a128936819a97ce29768c3e2e7a4a16224"} Nov 01 22:05:29.475815 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:29.475747 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" podStartSLOduration=2.475730668 podStartE2EDuration="2.475730668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:29.472302804 +0000 UTC m=+7337.146562061" watchObservedRunningTime="2026-11-01 22:05:29.475730668 +0000 UTC m=+7337.149989922" Nov 01 22:05:30.368959 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:30.368919 5633 generic.go:356] "Generic (PLEG): container finished" podID="7cfd804c-d0e8-4371-98dd-2d17f2a04f77" containerID="93a866cce35d2a790d294acf9a7fe3f484b70a3926cbe6316519d9ebfffae4b3" exitCode=0 Nov 01 22:05:30.369498 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:30.369005 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" event={"ID":"7cfd804c-d0e8-4371-98dd-2d17f2a04f77","Type":"ContainerDied","Data":"93a866cce35d2a790d294acf9a7fe3f484b70a3926cbe6316519d9ebfffae4b3"} Nov 01 22:05:31.553624 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.553588 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:31.660146 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.660044 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-openshift-ca\") pod \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " Nov 01 22:05:31.660146 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.660097 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-xks-ca\") pod \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " Nov 01 22:05:31.660490 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.660250 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-kube-api-access-qdzff\" (UniqueName: \"kubernetes.io/projected/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-kube-api-access-qdzff\") pod \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\" (UID: \"7cfd804c-d0e8-4371-98dd-2d17f2a04f77\") " Nov 01 22:05:31.660594 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.660563 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-openshift-ca" pod "7cfd804c-d0e8-4371-98dd-2d17f2a04f77" (UID: "7cfd804c-d0e8-4371-98dd-2d17f2a04f77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:31.663106 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.663074 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-xks-ca" pod "7cfd804c-d0e8-4371-98dd-2d17f2a04f77" (UID: "7cfd804c-d0e8-4371-98dd-2d17f2a04f77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:31.663106 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.663092 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-kube-api-access-qdzff" pod "7cfd804c-d0e8-4371-98dd-2d17f2a04f77" (UID: "7cfd804c-d0e8-4371-98dd-2d17f2a04f77"). InnerVolumeSpecName "kube-api-access-qdzff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:31.761708 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.761658 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:31.761708 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.761710 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:31.761945 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:31.761729 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdzff\" (UniqueName: \"kubernetes.io/projected/7cfd804c-d0e8-4371-98dd-2d17f2a04f77-kube-api-access-qdzff\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:32.384914 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:32.384879 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" Nov 01 22:05:32.384914 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:32.384901 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7642b99-nrhzq" event={"ID":"7cfd804c-d0e8-4371-98dd-2d17f2a04f77","Type":"ContainerDied","Data":"a279511afe3e69878eadcc9737eb22a128936819a97ce29768c3e2e7a4a16224"} Nov 01 22:05:32.385150 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:32.384950 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a279511afe3e69878eadcc9737eb22a128936819a97ce29768c3e2e7a4a16224" Nov 01 22:05:33.354359 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:33.352567 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fpzcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:05:33.354359 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:33.352655 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fpzcg" podUID="f31af8b8-3cb5-404c-b3ec-ecb67025655b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:33.564579 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:33.564527 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-x2nzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:05:33.564756 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:33.564593 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-x2nzz" podUID="42e570f1-73f4-456e-99de-2128fdbb9d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:33.752575 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:33.752538 5633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ltrbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:05:33.752972 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:33.752941 5633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ltrbx" podUID="bdbf238c-3422-47b0-92db-30d6379d92ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:38.743859 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.743817 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r"] Nov 01 22:05:38.744308 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.744295 5633 cpu_manager.go:414] "containerMap: removing container" podUID="7cfd804c-d0e8-4371-98dd-2d17f2a04f77" containerName="revoke-keys" Nov 01 22:05:38.744395 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.744308 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfd804c-d0e8-4371-98dd-2d17f2a04f77" containerName="revoke-keys" Nov 01 22:05:38.744448 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.744436 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfd804c-d0e8-4371-98dd-2d17f2a04f77" containerName="revoke-keys" Nov 01 22:05:38.747909 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.747891 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.803712 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.803682 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r"] Nov 01 22:05:38.847690 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.847650 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:38.847887 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.847739 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:38.847887 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.847762 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:05:38.849045 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.849028 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:38.852634 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.852608 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.852693 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.852657 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.852880 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.852858 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg778\" (UniqueName: \"kubernetes.io/projected/363b33bd-17ac-4db5-94db-920eb1f36699-kube-api-access-lg778\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.953663 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.953628 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.953850 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.953747 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg778\" (UniqueName: \"kubernetes.io/projected/363b33bd-17ac-4db5-94db-920eb1f36699-kube-api-access-lg778\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.953850 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.953787 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.954495 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.954428 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:38.956323 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:38.956294 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:39.010234 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:39.010161 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg778\" (UniqueName: \"kubernetes.io/projected/363b33bd-17ac-4db5-94db-920eb1f36699-kube-api-access-lg778\") pod \"maas-api-revoke-sub-bc510f09-h4t9r\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:39.060976 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:39.060945 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:39.436148 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:39.436109 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r"] Nov 01 22:05:39.436560 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:05:39.436534 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363b33bd_17ac_4db5_94db_920eb1f36699.slice/crio-2cc2c9f56dc78c05ce011c6754c2ad3da1f87230572a6e46619c968a34d9d685 WatchSource:0}: Error finding container 2cc2c9f56dc78c05ce011c6754c2ad3da1f87230572a6e46619c968a34d9d685: Status 404 returned error can't find the container with id 2cc2c9f56dc78c05ce011c6754c2ad3da1f87230572a6e46619c968a34d9d685 Nov 01 22:05:39.468793 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:39.468764 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" event={"ID":"363b33bd-17ac-4db5-94db-920eb1f36699","Type":"ContainerStarted","Data":"2cc2c9f56dc78c05ce011c6754c2ad3da1f87230572a6e46619c968a34d9d685"} Nov 01 22:05:40.475978 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:40.475943 5633 generic.go:356] "Generic (PLEG): container finished" podID="363b33bd-17ac-4db5-94db-920eb1f36699" containerID="7a8171b309a080a1ba1ad855ae7863300e376a5a9b7a1641431a3bba28a8bfe4" exitCode=0 Nov 01 22:05:40.476491 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:40.476020 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" event={"ID":"363b33bd-17ac-4db5-94db-920eb1f36699","Type":"ContainerDied","Data":"7a8171b309a080a1ba1ad855ae7863300e376a5a9b7a1641431a3bba28a8bfe4"} Nov 01 22:05:41.673989 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.673955 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:41.782291 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.782172 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-xks-ca\") pod \"363b33bd-17ac-4db5-94db-920eb1f36699\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " Nov 01 22:05:41.782499 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.782347 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-openshift-ca\") pod \"363b33bd-17ac-4db5-94db-920eb1f36699\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " Nov 01 22:05:41.782499 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.782446 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363b33bd-17ac-4db5-94db-920eb1f36699-kube-api-access-lg778\" (UniqueName: \"kubernetes.io/projected/363b33bd-17ac-4db5-94db-920eb1f36699-kube-api-access-lg778\") pod \"363b33bd-17ac-4db5-94db-920eb1f36699\" (UID: \"363b33bd-17ac-4db5-94db-920eb1f36699\") " Nov 01 22:05:41.783035 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.782997 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-openshift-ca" pod "363b33bd-17ac-4db5-94db-920eb1f36699" (UID: "363b33bd-17ac-4db5-94db-920eb1f36699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:41.806685 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.804231 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-xks-ca" pod "363b33bd-17ac-4db5-94db-920eb1f36699" (UID: "363b33bd-17ac-4db5-94db-920eb1f36699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:41.806685 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.806408 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363b33bd-17ac-4db5-94db-920eb1f36699-kube-api-access-lg778" pod "363b33bd-17ac-4db5-94db-920eb1f36699" (UID: "363b33bd-17ac-4db5-94db-920eb1f36699"). InnerVolumeSpecName "kube-api-access-lg778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:41.884171 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.884119 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg778\" (UniqueName: \"kubernetes.io/projected/363b33bd-17ac-4db5-94db-920eb1f36699-kube-api-access-lg778\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:41.884171 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.884171 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:41.884525 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:41.884187 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363b33bd-17ac-4db5-94db-920eb1f36699-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:42.493937 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:42.493904 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" Nov 01 22:05:42.493937 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:42.493916 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc510f09-h4t9r" event={"ID":"363b33bd-17ac-4db5-94db-920eb1f36699","Type":"ContainerDied","Data":"2cc2c9f56dc78c05ce011c6754c2ad3da1f87230572a6e46619c968a34d9d685"} Nov 01 22:05:42.494225 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:42.493963 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cc2c9f56dc78c05ce011c6754c2ad3da1f87230572a6e46619c968a34d9d685" Nov 01 22:05:46.156919 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:05:46.156867 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:05:49.044515 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.044478 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv"] Nov 01 22:05:49.093833 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.093796 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96a468d0-4xhvv"] Nov 01 22:05:49.527042 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.527003 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc"] Nov 01 22:05:49.527499 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.527485 5633 cpu_manager.go:414] "containerMap: removing container" podUID="363b33bd-17ac-4db5-94db-920eb1f36699" containerName="revoke-keys" Nov 01 22:05:49.527565 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.527501 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363b33bd-17ac-4db5-94db-920eb1f36699" containerName="revoke-keys" Nov 01 22:05:49.527609 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.527593 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="363b33bd-17ac-4db5-94db-920eb1f36699" containerName="revoke-keys" Nov 01 22:05:49.531028 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.531010 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.586880 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.586847 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc"] Nov 01 22:05:49.606142 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.606105 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwb4\" (UniqueName: \"kubernetes.io/projected/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-kube-api-access-tdwb4\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.606365 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.606248 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.606365 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.606316 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.631771 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.631728 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:05:49.631771 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.631742 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:05:49.631771 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.631763 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:05:49.632026 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.631825 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:05:49.707258 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.707215 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.707500 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.707296 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.707500 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.707415 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwb4\" (UniqueName: \"kubernetes.io/projected/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-kube-api-access-tdwb4\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.707959 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.707939 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.709882 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.709859 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.764254 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.764218 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwb4\" (UniqueName: \"kubernetes.io/projected/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-kube-api-access-tdwb4\") pod \"maas-api-revoke-sub-054e8a3c-vgjfc\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:49.844641 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:49.844549 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:50.221758 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:50.221731 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc"] Nov 01 22:05:50.222114 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:05:50.222082 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6b89fc_e1d4_4dd8_896f_0c5b8b1f9ff7.slice/crio-357b2c197b28fb758385fbfd428c118ebb686a150058fdce301470c6e2531762 WatchSource:0}: Error finding container 357b2c197b28fb758385fbfd428c118ebb686a150058fdce301470c6e2531762: Status 404 returned error can't find the container with id 357b2c197b28fb758385fbfd428c118ebb686a150058fdce301470c6e2531762 Nov 01 22:05:50.466923 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:50.466885 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa8a534-7cde-44c2-99c2-ff69a0bdc821" path="/var/lib/kubelet/pods/ffa8a534-7cde-44c2-99c2-ff69a0bdc821/volumes" Nov 01 22:05:50.593199 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:50.593156 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" event={"ID":"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7","Type":"ContainerStarted","Data":"85aff57dbcc64824e389fa27be5a7314cce54f62de09cce602912ce9df0da506"} Nov 01 22:05:50.593427 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:50.593220 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" event={"ID":"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7","Type":"ContainerStarted","Data":"357b2c197b28fb758385fbfd428c118ebb686a150058fdce301470c6e2531762"} Nov 01 22:05:50.704303 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:50.704243 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" podStartSLOduration=1.704227667 podStartE2EDuration="1.704227667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:50.704000215 +0000 UTC m=+7358.378259472" watchObservedRunningTime="2026-11-01 22:05:50.704227667 +0000 UTC m=+7358.378486915" Nov 01 22:05:51.601133 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:51.601097 5633 generic.go:356] "Generic (PLEG): container finished" podID="ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7" containerID="85aff57dbcc64824e389fa27be5a7314cce54f62de09cce602912ce9df0da506" exitCode=0 Nov 01 22:05:51.601579 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:51.601185 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" event={"ID":"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7","Type":"ContainerDied","Data":"85aff57dbcc64824e389fa27be5a7314cce54f62de09cce602912ce9df0da506"} Nov 01 22:05:52.785828 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.785797 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:52.836903 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.836856 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-xks-ca\") pod \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " Nov 01 22:05:52.837125 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.836953 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-kube-api-access-tdwb4\" (UniqueName: \"kubernetes.io/projected/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-kube-api-access-tdwb4\") pod \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " Nov 01 22:05:52.837125 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.837030 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-openshift-ca\") pod \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\" (UID: \"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7\") " Nov 01 22:05:52.837752 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.837703 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-openshift-ca" pod "ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7" (UID: "ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:52.839480 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.839449 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-kube-api-access-tdwb4" pod "ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7" (UID: "ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7"). InnerVolumeSpecName "kube-api-access-tdwb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:52.839839 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.839815 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-xks-ca" pod "ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7" (UID: "ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:52.938438 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.938239 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:52.938438 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.938277 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdwb4\" (UniqueName: \"kubernetes.io/projected/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-kube-api-access-tdwb4\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:52.938438 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:52.938287 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:05:53.651767 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:53.651727 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" event={"ID":"ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7","Type":"ContainerDied","Data":"357b2c197b28fb758385fbfd428c118ebb686a150058fdce301470c6e2531762"} Nov 01 22:05:53.652068 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:53.652047 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="357b2c197b28fb758385fbfd428c118ebb686a150058fdce301470c6e2531762" Nov 01 22:05:53.652263 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:53.652247 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054e8a3c-vgjfc" Nov 01 22:05:59.626611 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:05:59.626570 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:00.046495 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.046457 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t"] Nov 01 22:06:00.095702 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.095655 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a14a6aad-qns8t"] Nov 01 22:06:00.266784 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.266749 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86"] Nov 01 22:06:00.267225 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.267212 5633 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7" containerName="revoke-keys" Nov 01 22:06:00.267271 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.267228 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7" containerName="revoke-keys" Nov 01 22:06:00.267360 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.267350 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7" containerName="revoke-keys" Nov 01 22:06:00.270790 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.270774 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.323311 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.323203 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:06:00.323311 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.323281 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:06:00.323694 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.323670 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:06:00.323787 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.323718 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:06:00.327712 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.327685 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86"] Nov 01 22:06:00.332273 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.332242 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.332436 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.332299 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.332556 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.332532 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph867\" (UniqueName: \"kubernetes.io/projected/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-kube-api-access-ph867\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.433961 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.433921 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph867\" (UniqueName: \"kubernetes.io/projected/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-kube-api-access-ph867\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.434178 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.434015 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.434178 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.434039 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.434799 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.434770 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.436650 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.436629 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.466870 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.466837 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03" path="/var/lib/kubelet/pods/bcc4afd2-1b18-4324-b9ba-a0bfd8b5ba03/volumes" Nov 01 22:06:00.491061 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.491026 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph867\" (UniqueName: \"kubernetes.io/projected/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-kube-api-access-ph867\") pod \"maas-api-revoke-sub-4f622021-5xb86\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.584240 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.584151 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:00.963117 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:00.963093 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86"] Nov 01 22:06:00.964866 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:06:00.964840 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff972cf_f762_4387_bdaa_c6bb3b674cd4.slice/crio-f2168ffbd3f4040fcd8426d6a2d2c87434be738d5fa23e182a4abb9e2147bd57 WatchSource:0}: Error finding container f2168ffbd3f4040fcd8426d6a2d2c87434be738d5fa23e182a4abb9e2147bd57: Status 404 returned error can't find the container with id f2168ffbd3f4040fcd8426d6a2d2c87434be738d5fa23e182a4abb9e2147bd57 Nov 01 22:06:01.720817 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:01.720777 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" event={"ID":"0ff972cf-f762-4387-bdaa-c6bb3b674cd4","Type":"ContainerStarted","Data":"b0ec2f93e954beb660fd01848a93c5307fe24af8ee56bc585b4bc7398c95d94f"} Nov 01 22:06:01.721014 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:01.720843 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" event={"ID":"0ff972cf-f762-4387-bdaa-c6bb3b674cd4","Type":"ContainerStarted","Data":"f2168ffbd3f4040fcd8426d6a2d2c87434be738d5fa23e182a4abb9e2147bd57"} Nov 01 22:06:01.837901 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:01.837847 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" podStartSLOduration=1.837829696 podStartE2EDuration="1.837829696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:01.83654136 +0000 UTC m=+7369.510800615" watchObservedRunningTime="2026-11-01 22:06:01.837829696 +0000 UTC m=+7369.512088950" Nov 01 22:06:02.728324 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:02.728288 5633 generic.go:356] "Generic (PLEG): container finished" podID="0ff972cf-f762-4387-bdaa-c6bb3b674cd4" containerID="b0ec2f93e954beb660fd01848a93c5307fe24af8ee56bc585b4bc7398c95d94f" exitCode=0 Nov 01 22:06:02.728764 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:02.728380 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" event={"ID":"0ff972cf-f762-4387-bdaa-c6bb3b674cd4","Type":"ContainerDied","Data":"b0ec2f93e954beb660fd01848a93c5307fe24af8ee56bc585b4bc7398c95d94f"} Nov 01 22:06:03.910522 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.910491 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:03.971765 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.971728 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-openshift-ca\") pod \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " Nov 01 22:06:03.971976 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.971941 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-xks-ca\") pod \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " Nov 01 22:06:03.972081 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.972010 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-kube-api-access-ph867\" (UniqueName: \"kubernetes.io/projected/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-kube-api-access-ph867\") pod \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\" (UID: \"0ff972cf-f762-4387-bdaa-c6bb3b674cd4\") " Nov 01 22:06:03.972177 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.972144 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-openshift-ca" pod "0ff972cf-f762-4387-bdaa-c6bb3b674cd4" (UID: "0ff972cf-f762-4387-bdaa-c6bb3b674cd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:03.972430 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.972404 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:06:03.974709 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.974673 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-xks-ca" pod "0ff972cf-f762-4387-bdaa-c6bb3b674cd4" (UID: "0ff972cf-f762-4387-bdaa-c6bb3b674cd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:06:03.974976 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:03.974946 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-kube-api-access-ph867" pod "0ff972cf-f762-4387-bdaa-c6bb3b674cd4" (UID: "0ff972cf-f762-4387-bdaa-c6bb3b674cd4"). InnerVolumeSpecName "kube-api-access-ph867". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:04.074545 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:04.074435 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:06:04.074545 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:04.074477 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph867\" (UniqueName: \"kubernetes.io/projected/0ff972cf-f762-4387-bdaa-c6bb3b674cd4-kube-api-access-ph867\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:06:04.759361 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:04.759233 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" Nov 01 22:06:04.760448 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:04.760155 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f622021-5xb86" event={"ID":"0ff972cf-f762-4387-bdaa-c6bb3b674cd4","Type":"ContainerDied","Data":"f2168ffbd3f4040fcd8426d6a2d2c87434be738d5fa23e182a4abb9e2147bd57"} Nov 01 22:06:04.760448 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:04.760213 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2168ffbd3f4040fcd8426d6a2d2c87434be738d5fa23e182a4abb9e2147bd57" Nov 01 22:06:10.989454 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:10.989420 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm"] Nov 01 22:06:10.990136 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:10.990114 5633 cpu_manager.go:414] "containerMap: removing container" podUID="0ff972cf-f762-4387-bdaa-c6bb3b674cd4" containerName="revoke-keys" Nov 01 22:06:10.990260 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:10.990139 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff972cf-f762-4387-bdaa-c6bb3b674cd4" containerName="revoke-keys" Nov 01 22:06:10.990260 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:10.990257 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff972cf-f762-4387-bdaa-c6bb3b674cd4" containerName="revoke-keys" Nov 01 22:06:10.993570 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:10.993548 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.050002 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.049969 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm"] Nov 01 22:06:11.091204 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.091168 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk69s\" (UniqueName: \"kubernetes.io/projected/3196eeca-7332-476a-aafa-18af16225b0d-kube-api-access-kk69s\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.091413 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.091309 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.091413 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.091376 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.094570 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.094549 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:06:11.094637 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.094556 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:06:11.095670 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.095643 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\"" Nov 01 22:06:11.095834 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.095677 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:06:11.192480 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.192441 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.192706 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.192502 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk69s\" (UniqueName: \"kubernetes.io/projected/3196eeca-7332-476a-aafa-18af16225b0d-kube-api-access-kk69s\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.192706 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.192580 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.193201 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.193175 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.195067 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.195046 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.248446 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.248365 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk69s\" (UniqueName: \"kubernetes.io/projected/3196eeca-7332-476a-aafa-18af16225b0d-kube-api-access-kk69s\") pod \"maas-api-revoke-sub-88d34b1b-fbcvm\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.311489 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.311449 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:11.692094 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.692064 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm"] Nov 01 22:06:11.692713 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:06:11.692684 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3196eeca_7332_476a_aafa_18af16225b0d.slice/crio-c2aabff9482c12cd325bf83d982eac009e1f8beb437d51d614396f2de0f6c57d WatchSource:0}: Error finding container c2aabff9482c12cd325bf83d982eac009e1f8beb437d51d614396f2de0f6c57d: Status 404 returned error can't find the container with id c2aabff9482c12cd325bf83d982eac009e1f8beb437d51d614396f2de0f6c57d Nov 01 22:06:11.806509 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:11.806471 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" event={"ID":"3196eeca-7332-476a-aafa-18af16225b0d","Type":"ContainerStarted","Data":"c2aabff9482c12cd325bf83d982eac009e1f8beb437d51d614396f2de0f6c57d"} Nov 01 22:06:12.813899 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:12.813811 5633 generic.go:356] "Generic (PLEG): container finished" podID="3196eeca-7332-476a-aafa-18af16225b0d" containerID="a12b5a922a6513f12c4795d05c09f578b49627004c1ac2bf7efd1ad3ef2f6ab6" exitCode=0 Nov 01 22:06:12.814283 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:12.813901 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" event={"ID":"3196eeca-7332-476a-aafa-18af16225b0d","Type":"ContainerDied","Data":"a12b5a922a6513f12c4795d05c09f578b49627004c1ac2bf7efd1ad3ef2f6ab6"} Nov 01 22:06:14.263927 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.263576 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:14.480356 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.476451 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-xks-ca\") pod \"3196eeca-7332-476a-aafa-18af16225b0d\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " Nov 01 22:06:14.480356 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.476609 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-openshift-ca\") pod \"3196eeca-7332-476a-aafa-18af16225b0d\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " Nov 01 22:06:14.480356 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.476650 5633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3196eeca-7332-476a-aafa-18af16225b0d-kube-api-access-kk69s\" (UniqueName: \"kubernetes.io/projected/3196eeca-7332-476a-aafa-18af16225b0d-kube-api-access-kk69s\") pod \"3196eeca-7332-476a-aafa-18af16225b0d\" (UID: \"3196eeca-7332-476a-aafa-18af16225b0d\") " Nov 01 22:06:14.480725 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.480509 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-openshift-ca" pod "3196eeca-7332-476a-aafa-18af16225b0d" (UID: "3196eeca-7332-476a-aafa-18af16225b0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:14.483563 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.483526 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-xks-ca" pod "3196eeca-7332-476a-aafa-18af16225b0d" (UID: "3196eeca-7332-476a-aafa-18af16225b0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:06:14.484538 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.484505 5633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3196eeca-7332-476a-aafa-18af16225b0d-kube-api-access-kk69s" pod "3196eeca-7332-476a-aafa-18af16225b0d" (UID: "3196eeca-7332-476a-aafa-18af16225b0d"). InnerVolumeSpecName "kube-api-access-kk69s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:14.577992 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.577962 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-openshift-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.577992 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.577989 5633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk69s\" (UniqueName: \"kubernetes.io/projected/3196eeca-7332-476a-aafa-18af16225b0d-kube-api-access-kk69s\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.578201 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.578000 5633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196eeca-7332-476a-aafa-18af16225b0d-maas-api-xks-ca\") on node \"ip-10-0-128-78.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.654284 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.654179 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:14.824400 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.824275 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:14.864450 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.862476 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" event={"ID":"3196eeca-7332-476a-aafa-18af16225b0d","Type":"ContainerDied","Data":"c2aabff9482c12cd325bf83d982eac009e1f8beb437d51d614396f2de0f6c57d"} Nov 01 22:06:14.864450 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.862539 5633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2aabff9482c12cd325bf83d982eac009e1f8beb437d51d614396f2de0f6c57d" Nov 01 22:06:14.864450 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:14.862614 5633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d34b1b-fbcvm" Nov 01 22:06:16.174644 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:06:16.174601 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:06:23.787081 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:23.786983 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ld4qw"] Nov 01 22:06:30.998651 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:30.998613 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:31.160981 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:31.160925 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:06:34.049821 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:34.049785 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg"] Nov 01 22:06:34.101547 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:34.101504 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cf080-4mnqg"] Nov 01 22:06:34.468892 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:34.468851 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9cb3f6-2c39-431f-963e-08b4afeaf953" path="/var/lib/kubelet/pods/4c9cb3f6-2c39-431f-963e-08b4afeaf953/volumes" Nov 01 22:06:42.444109 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:42.444074 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cf654df49-4ch47_f1ef9e76-0e8b-4050-b269-5042540f7826/authorino/0.log" Nov 01 22:06:46.158957 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:06:46.158908 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:06:53.529978 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:53.529937 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/1.log" Nov 01 22:06:53.874048 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:53.873961 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/1.log" Nov 01 22:06:55.179440 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:55.179405 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645c58c6f7-fvkrq_3249de5b-b781-4eeb-8fe0-9bf4455e9af1/manager/0.log" Nov 01 22:06:55.463993 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:55.463968 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nscg4_775c372c-52f9-4571-b7c1-28905e6f854c/server/0.log" Nov 01 22:06:55.672274 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:55.672235 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wpr7v_d5778aa7-6472-4e46-afa1-4048cb03ef96/manager/0.log" Nov 01 22:06:55.921590 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:55.921507 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjv47_292d7509-aef9-452f-ab7f-85e1117d9652/proxy/0.log" Nov 01 22:06:56.542346 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:56.542299 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/0.log" Nov 01 22:06:56.762309 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:56.762276 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/0.log" Nov 01 22:06:57.929935 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:57.929903 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-028c9594-t4r7n_9835e43d-f139-469b-9eed-1d67e958512b/revoke-keys/0.log" Nov 01 22:06:58.148248 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:58.148212 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-054e8a3c-vgjfc_ff6b89fc-e1d4-4dd8-896f-0c5b8b1f9ff7/revoke-keys/0.log" Nov 01 22:06:58.355972 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:58.355937 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0688a8f2-j8qgq_4c1f2d12-8aca-4688-bbbf-3925080d041c/revoke-keys/0.log" Nov 01 22:06:58.577114 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:58.577064 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0eb8e4bd-mmgvs_ab564d94-16e6-49bc-a2d1-018c50d7cfde/revoke-keys/0.log" Nov 01 22:06:58.790897 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:58.790862 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2afc9869-sffhc_29383b31-0598-4821-8671-89805998a3ee/revoke-keys/0.log" Nov 01 22:06:59.007740 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:59.007705 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4f622021-5xb86_0ff972cf-f762-4387-bdaa-c6bb3b674cd4/revoke-keys/0.log" Nov 01 22:06:59.205710 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:59.205616 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-54a5177f-spfwm_3df73ce0-9e69-4988-bb01-7cfc33850ba1/revoke-keys/0.log" Nov 01 22:06:59.416252 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:59.416221 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-75a6d999-828tq_4d0578d5-b13f-4b5f-b6c5-88876a70bc98/revoke-keys/0.log" Nov 01 22:06:59.626515 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:59.626482 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-887d0aa3-plr8w_6c10247a-e55b-420e-9829-c9584b5afa3e/revoke-keys/0.log" Nov 01 22:06:59.819522 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:59.819495 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-88d34b1b-fbcvm_3196eeca-7332-476a-aafa-18af16225b0d/revoke-keys/0.log" Nov 01 22:06:59.945298 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:06:59.945207 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:00.030861 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:00.030828 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9e9f5dbb-8j8sw_c539f5a6-0705-4f51-b1a5-7179c92aa7e4/revoke-keys/0.log" Nov 01 22:07:00.248754 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:00.248722 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a7642b99-nrhzq_7cfd804c-d0e8-4371-98dd-2d17f2a04f77/revoke-keys/0.log" Nov 01 22:07:00.455440 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:00.455411 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab8d7e49-4kdcd_37bee66e-427e-4db4-9112-3256841d368c/revoke-keys/0.log" Nov 01 22:07:00.670205 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:00.670117 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bc510f09-h4t9r_363b33bd-17ac-4db5-94db-920eb1f36699/revoke-keys/0.log" Nov 01 22:07:00.875060 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:00.875032 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ccc1ffd1-nxgdh_79533546-45d3-4a69-9fb1-a37450126c75/revoke-keys/0.log" Nov 01 22:07:01.073567 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:01.073536 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f201f7fe-nwswt_fcb92700-3f1c-46f1-a2f2-67f9fa501d8b/revoke-keys/0.log" Nov 01 22:07:03.116626 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:03.116593 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cf654df49-4ch47_f1ef9e76-0e8b-4050-b269-5042540f7826/authorino/0.log" Nov 01 22:07:03.821785 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:03.821739 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/2.log" Nov 01 22:07:05.122139 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:05.122104 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ld4qw_f956c89a-217e-4237-a2cb-70632240750d/limitador/0.log" Nov 01 22:07:05.349404 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:05.349367 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wdb8s_a8e46f26-efa8-4823-b684-b805cce7bf88/manager/0.log" Nov 01 22:07:06.131586 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:06.131552 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/1.log" Nov 01 22:07:07.281791 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:07.281753 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c84dcdb75-9t9pp_1c67a118-8b19-4fe3-8a24-71875f068830/kube-auth-proxy/0.log" Nov 01 22:07:08.408475 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:08.408445 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fpzcg_f31af8b8-3cb5-404c-b3ec-ecb67025655b/istio-proxy/0.log" Nov 01 22:07:08.847499 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:08.847468 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ltrbx_bdbf238c-3422-47b0-92db-30d6379d92ec/istio-proxy/0.log" Nov 01 22:07:09.733555 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:09.733528 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-x2nzz_42e570f1-73f4-456e-99de-2128fdbb9d4e/istio-proxy/0.log" Nov 01 22:07:11.891962 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:11.891929 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-v9p6m_ae6287c3-1095-4e2e-9722-8731aa3a8235/main/0.log" Nov 01 22:07:12.159241 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:12.159155 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vkc6h_fcd4b73a-baf7-471d-8129-73c6934027ee/main/0.log" Nov 01 22:07:12.421619 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:12.421544 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jgzgn_dbc07a3f-9280-4bf8-aa49-b59a3a4383a9/main/0.log" Nov 01 22:07:12.857859 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:12.857831 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4v5f2_c299f460-9ff7-4b2d-ba42-50d6d4a5e810/main/0.log" Nov 01 22:07:16.046481 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:16.046446 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq"] Nov 01 22:07:16.095844 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:16.095801 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a6d999-828tq"] Nov 01 22:07:16.156357 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:07:16.156294 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:07:16.466764 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:16.466722 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0578d5-b13f-4b5f-b6c5-88876a70bc98" path="/var/lib/kubelet/pods/4d0578d5-b13f-4b5f-b6c5-88876a70bc98/volumes" Nov 01 22:07:19.590938 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:19.590902 5633 scope.go:122] "RemoveContainer" containerID="bec303896bd5f8ba4e4811717380cede39368dd1d2e12c012cc2c834a83dd942" Nov 01 22:07:19.601019 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:19.600997 5633 scope.go:122] "RemoveContainer" containerID="ddb9dd13e8001093db2d263b5759dba26bd0f7bbf4e99a3f8a825edc108f4e22" Nov 01 22:07:23.256058 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.256016 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4v85z/must-gather-dxc7x"] Nov 01 22:07:23.256766 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.256744 5633 cpu_manager.go:414] "containerMap: removing container" podUID="3196eeca-7332-476a-aafa-18af16225b0d" containerName="revoke-keys" Nov 01 22:07:23.256867 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.256768 5633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3196eeca-7332-476a-aafa-18af16225b0d" containerName="revoke-keys" Nov 01 22:07:23.256931 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.256882 5633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3196eeca-7332-476a-aafa-18af16225b0d" containerName="revoke-keys" Nov 01 22:07:23.261887 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.261861 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.269436 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.269400 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:23.354971 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.354942 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4v85z\"/\"kube-root-ca.crt\"" Nov 01 22:07:23.362584 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.362558 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4v85z\"/\"default-dockercfg-6hg82\"" Nov 01 22:07:23.362584 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.362558 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4v85z\"/\"openshift-service-ca.crt\"" Nov 01 22:07:23.366371 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.366344 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4v85z/must-gather-dxc7x"] Nov 01 22:07:23.378515 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.378485 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73387b28-a062-4b4c-acbf-c0c6b44864f7-must-gather-output\") pod \"must-gather-dxc7x\" (UID: \"73387b28-a062-4b4c-acbf-c0c6b44864f7\") " pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.378698 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.378527 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zff\" (UniqueName: \"kubernetes.io/projected/73387b28-a062-4b4c-acbf-c0c6b44864f7-kube-api-access-62zff\") pod \"must-gather-dxc7x\" (UID: \"73387b28-a062-4b4c-acbf-c0c6b44864f7\") " pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.401561 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.401525 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:07:23.480029 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.479988 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73387b28-a062-4b4c-acbf-c0c6b44864f7-must-gather-output\") pod \"must-gather-dxc7x\" (UID: \"73387b28-a062-4b4c-acbf-c0c6b44864f7\") " pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.480225 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.480034 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zff\" (UniqueName: \"kubernetes.io/projected/73387b28-a062-4b4c-acbf-c0c6b44864f7-kube-api-access-62zff\") pod \"must-gather-dxc7x\" (UID: \"73387b28-a062-4b4c-acbf-c0c6b44864f7\") " pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.480397 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.480368 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73387b28-a062-4b4c-acbf-c0c6b44864f7-must-gather-output\") pod \"must-gather-dxc7x\" (UID: \"73387b28-a062-4b4c-acbf-c0c6b44864f7\") " pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.539186 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.539107 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zff\" (UniqueName: \"kubernetes.io/projected/73387b28-a062-4b4c-acbf-c0c6b44864f7-kube-api-access-62zff\") pod \"must-gather-dxc7x\" (UID: \"73387b28-a062-4b4c-acbf-c0c6b44864f7\") " pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.578044 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.578008 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4v85z/must-gather-dxc7x" Nov 01 22:07:23.952187 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:07:23.952156 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73387b28_a062_4b4c_acbf_c0c6b44864f7.slice/crio-c29e1f141aaa8fb50cc572f5852a373890d8160c4fee0ea0bb4fe91055869495 WatchSource:0}: Error finding container c29e1f141aaa8fb50cc572f5852a373890d8160c4fee0ea0bb4fe91055869495: Status 404 returned error can't find the container with id c29e1f141aaa8fb50cc572f5852a373890d8160c4fee0ea0bb4fe91055869495 Nov 01 22:07:23.952628 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.952602 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4v85z/must-gather-dxc7x"] Nov 01 22:07:23.954267 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:23.954252 5633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:07:24.358027 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:24.357987 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4v85z/must-gather-dxc7x" event={"ID":"73387b28-a062-4b4c-acbf-c0c6b44864f7","Type":"ContainerStarted","Data":"c29e1f141aaa8fb50cc572f5852a373890d8160c4fee0ea0bb4fe91055869495"} Nov 01 22:07:27.048232 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:27.048187 5633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w"] Nov 01 22:07:27.095842 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:27.095810 5633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887d0aa3-plr8w"] Nov 01 22:07:28.467858 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:28.467826 5633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c10247a-e55b-420e-9829-c9584b5afa3e" path="/var/lib/kubelet/pods/6c10247a-e55b-420e-9829-c9584b5afa3e/volumes" Nov 01 22:07:31.416278 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:31.416224 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4v85z/must-gather-dxc7x" event={"ID":"73387b28-a062-4b4c-acbf-c0c6b44864f7","Type":"ContainerStarted","Data":"b4e0bb24516c17420e7ed4b7a4534a7a5803ab3285d034a6a103c1ac7b4a3f3e"} Nov 01 22:07:31.416278 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:31.416282 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4v85z/must-gather-dxc7x" event={"ID":"73387b28-a062-4b4c-acbf-c0c6b44864f7","Type":"ContainerStarted","Data":"7eb071b0475ca2982c534bc0b74e93d4450321583a0878b744dfae85ae9746e9"} Nov 01 22:07:31.532740 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:31.532683 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4v85z/must-gather-dxc7x" podStartSLOduration=1.5252526899999999 podStartE2EDuration="8.532666315s" totalImagesPullingTime="7.007413625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:31.528293535 +0000 UTC m=+7459.202552790" watchObservedRunningTime="2026-11-01 22:07:31.532666315 +0000 UTC m=+7459.206925570" Nov 01 22:07:32.350831 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:32.350796 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7lwmf_7a6b3e1d-ddca-43d2-b5c4-300dcacd99ee/global-pull-secret-syncer/0.log" Nov 01 22:07:35.117780 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:35.117749 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/0.log" Nov 01 22:07:35.135528 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:35.135494 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/1.log" Nov 01 22:07:35.376937 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:35.376838 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/1.log" Nov 01 22:07:35.376937 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:35.376887 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/0.log" Nov 01 22:07:35.481253 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:35.481212 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" Nov 01 22:07:36.245886 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:36.245850 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645c58c6f7-fvkrq_3249de5b-b781-4eeb-8fe0-9bf4455e9af1/manager/0.log" Nov 01 22:07:36.353686 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:36.353658 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nscg4_775c372c-52f9-4571-b7c1-28905e6f854c/server/0.log" Nov 01 22:07:36.708631 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:36.708532 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-78.us-west-1.compute.internal_c04d0eff16222e7baef223e711863fea/haproxy/0.log" Nov 01 22:07:37.774432 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:37.774398 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wpr7v_d5778aa7-6472-4e46-afa1-4048cb03ef96/manager/0.log" Nov 01 22:07:37.901922 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:37.901891 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjv47_292d7509-aef9-452f-ab7f-85e1117d9652/proxy/0.log" Nov 01 22:07:40.719187 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:40.719148 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/0.log" Nov 01 22:07:40.737256 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:40.737226 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/1.log" Nov 01 22:07:40.979069 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:40.978979 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/1.log" Nov 01 22:07:40.979242 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:40.979069 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/0.log" Nov 01 22:07:41.415861 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:41.415782 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cf654df49-4ch47_f1ef9e76-0e8b-4050-b269-5042540f7826/authorino/0.log" Nov 01 22:07:41.804233 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:41.804192 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645c58c6f7-fvkrq_3249de5b-b781-4eeb-8fe0-9bf4455e9af1/manager/0.log" Nov 01 22:07:41.937099 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:41.937071 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nscg4_775c372c-52f9-4571-b7c1-28905e6f854c/server/0.log" Nov 01 22:07:42.594031 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.593970 5633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh"] Nov 01 22:07:42.599005 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.598975 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.653896 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.653858 5633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h9v88\"/\"default-dockercfg-rbhbt\"" Nov 01 22:07:42.705100 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.705045 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h9v88\"/\"kube-root-ca.crt\"" Nov 01 22:07:42.705320 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.705297 5633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h9v88\"/\"openshift-service-ca.crt\"" Nov 01 22:07:42.710089 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.710056 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh"] Nov 01 22:07:42.786396 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.786358 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-proc\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.786654 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.786413 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfzj\" (UniqueName: \"kubernetes.io/projected/65fed043-32f8-4de8-829d-591b3377a5ea-kube-api-access-xzfzj\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.786654 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.786507 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-sys\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.786654 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.786545 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-podres\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.786654 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.786562 5633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-lib-modules\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.887703 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.887614 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-sys\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.888252 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.887743 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-podres\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.888252 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.887775 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-lib-modules\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.888252 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.887911 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-sys\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.888252 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.888039 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-lib-modules\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.888641 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.888271 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-proc\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.888641 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.888416 5633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfzj\" (UniqueName: \"kubernetes.io/projected/65fed043-32f8-4de8-829d-591b3377a5ea-kube-api-access-xzfzj\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.888641 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.888502 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-podres\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.889283 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.889260 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65fed043-32f8-4de8-829d-591b3377a5ea-proc\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:42.935522 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.935490 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wpr7v_d5778aa7-6472-4e46-afa1-4048cb03ef96/manager/0.log" Nov 01 22:07:42.946940 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:42.946896 5633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfzj\" (UniqueName: \"kubernetes.io/projected/65fed043-32f8-4de8-829d-591b3377a5ea-kube-api-access-xzfzj\") pod \"perf-node-gather-daemonset-klchh\" (UID: \"65fed043-32f8-4de8-829d-591b3377a5ea\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:43.125347 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:43.125303 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjv47_292d7509-aef9-452f-ab7f-85e1117d9652/proxy/0.log" Nov 01 22:07:43.224953 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:43.224921 5633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:43.506022 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:43.505946 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/1.log" Nov 01 22:07:43.506468 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:43.506443 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/2.log" Nov 01 22:07:43.614115 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:43.614086 5633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh"] Nov 01 22:07:43.614499 ip-10-0-128-78 kubenswrapper[5633]: W1101 22:07:43.614477 5633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod65fed043_32f8_4de8_829d_591b3377a5ea.slice/crio-04663e7ee6dc779f85a1323e9b58976aa7199fb0604741b842404d5635fa8d24 WatchSource:0}: Error finding container 04663e7ee6dc779f85a1323e9b58976aa7199fb0604741b842404d5635fa8d24: Status 404 returned error can't find the container with id 04663e7ee6dc779f85a1323e9b58976aa7199fb0604741b842404d5635fa8d24 Nov 01 22:07:44.508051 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:44.508014 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" event={"ID":"65fed043-32f8-4de8-829d-591b3377a5ea","Type":"ContainerStarted","Data":"af2b2056731d8ea324b998d4783af4eb6fa5a77d809a1c65a18b1e82a428d055"} Nov 01 22:07:44.508574 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:44.508080 5633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" event={"ID":"65fed043-32f8-4de8-829d-591b3377a5ea","Type":"ContainerStarted","Data":"04663e7ee6dc779f85a1323e9b58976aa7199fb0604741b842404d5635fa8d24"} Nov 01 22:07:44.508574 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:44.508221 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:44.622873 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:44.622815 5633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" podStartSLOduration=2.622801089 podStartE2EDuration="2.622801089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:44.621156117 +0000 UTC m=+7472.295415371" watchObservedRunningTime="2026-11-01 22:07:44.622801089 +0000 UTC m=+7472.297060361" Nov 01 22:07:44.929759 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:44.929671 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ld4qw_f956c89a-217e-4237-a2cb-70632240750d/limitador/0.log" Nov 01 22:07:45.515890 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:45.515840 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wdb8s_a8e46f26-efa8-4823-b684-b805cce7bf88/manager/0.log" Nov 01 22:07:45.990465 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:45.990428 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/0.log" Nov 01 22:07:46.008786 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:46.008756 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d6db5d4f-6rv48_2e038c41-6c0d-45d8-8fcd-2c8ac51ba295/manager/1.log" Nov 01 22:07:46.159458 ip-10-0-128-78 kubenswrapper[5633]: E1101 22:07:46.159404 5633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwwcz" containerName="extractor" Nov 01 22:07:46.197998 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:46.197968 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/1.log" Nov 01 22:07:46.198220 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:46.198068 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4dzk7_ae182c81-580e-4e54-bf4b-9696750c18ed/manager/0.log" Nov 01 22:07:47.152871 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:47.152836 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645c58c6f7-fvkrq_3249de5b-b781-4eeb-8fe0-9bf4455e9af1/manager/0.log" Nov 01 22:07:47.323163 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:47.323136 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nscg4_775c372c-52f9-4571-b7c1-28905e6f854c/server/0.log" Nov 01 22:07:48.051249 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:48.051212 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_519cd664-61e6-4b51-bd44-8a3bac94dbe1/alertmanager/0.log" Nov 01 22:07:48.502981 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:48.502946 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_519cd664-61e6-4b51-bd44-8a3bac94dbe1/config-reloader/0.log" Nov 01 22:07:48.694083 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:48.694040 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wpr7v_d5778aa7-6472-4e46-afa1-4048cb03ef96/manager/0.log" Nov 01 22:07:48.855837 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:48.855750 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjv47_292d7509-aef9-452f-ab7f-85e1117d9652/proxy/0.log" Nov 01 22:07:48.965042 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:48.965016 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_519cd664-61e6-4b51-bd44-8a3bac94dbe1/kube-rbac-proxy-web/0.log" Nov 01 22:07:49.381007 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:49.380973 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_519cd664-61e6-4b51-bd44-8a3bac94dbe1/kube-rbac-proxy/0.log" Nov 01 22:07:49.796691 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:49.796658 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_519cd664-61e6-4b51-bd44-8a3bac94dbe1/kube-rbac-proxy-metric/0.log" Nov 01 22:07:50.213126 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:50.213092 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_519cd664-61e6-4b51-bd44-8a3bac94dbe1/prom-label-proxy/0.log" Nov 01 22:07:50.530656 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:50.529655 5633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-klchh" Nov 01 22:07:50.630950 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:50.630914 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_519cd664-61e6-4b51-bd44-8a3bac94dbe1/init-config-reloader/0.log" Nov 01 22:07:51.481898 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:51.481813 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-586nr_f2c85daa-8760-4950-8cb2-e07c2aa5292e/kube-state-metrics/0.log" Nov 01 22:07:51.890615 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:51.890531 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-586nr_f2c85daa-8760-4950-8cb2-e07c2aa5292e/kube-rbac-proxy-main/0.log" Nov 01 22:07:52.305704 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:52.305675 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-586nr_f2c85daa-8760-4950-8cb2-e07c2aa5292e/kube-rbac-proxy-self/0.log" Nov 01 22:07:52.726564 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:52.726524 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d569ccc7c-67pl4_853d6d5f-e795-45b9-b633-48f2f0051ffc/metrics-server/0.log" Nov 01 22:07:53.568744 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:53.568704 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hnrf_4931755e-e9dc-4a2f-aab7-5d64dacb361e/node-exporter/0.log" Nov 01 22:07:53.984622 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:53.984592 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hnrf_4931755e-e9dc-4a2f-aab7-5d64dacb361e/kube-rbac-proxy/0.log" Nov 01 22:07:54.401126 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:54.401047 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hnrf_4931755e-e9dc-4a2f-aab7-5d64dacb361e/init-textfile/0.log" Nov 01 22:07:55.006089 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:55.006062 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c84dcdb75-9t9pp_1c67a118-8b19-4fe3-8a24-71875f068830/kube-auth-proxy/0.log" Nov 01 22:07:56.145195 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:56.145159 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fpzcg_f31af8b8-3cb5-404c-b3ec-ecb67025655b/istio-proxy/0.log" Nov 01 22:07:56.567244 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:56.567198 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ltrbx_bdbf238c-3422-47b0-92db-30d6379d92ec/istio-proxy/0.log" Nov 01 22:07:57.413281 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:57.413237 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-x2nzz_42e570f1-73f4-456e-99de-2128fdbb9d4e/istio-proxy/0.log" Nov 01 22:07:58.599741 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:58.599710 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6/prometheus/0.log" Nov 01 22:07:59.013664 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:59.013629 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6/config-reloader/0.log" Nov 01 22:07:59.430071 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:59.429994 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6/thanos-sidecar/0.log" Nov 01 22:07:59.847100 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:07:59.847064 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6/kube-rbac-proxy-web/0.log" Nov 01 22:08:00.263743 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:00.263704 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6/kube-rbac-proxy/0.log" Nov 01 22:08:00.288296 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:00.288257 5633 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8rbsz_d979ea87-f591-49d1-a326-bb58c0af72aa/konnectivity-agent/0.log" limit=1024 Nov 01 22:08:00.680776 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:00.680689 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6/kube-rbac-proxy-thanos/0.log" Nov 01 22:08:01.098126 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:01.098102 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4407a2d2-08d0-4c4a-b8bd-509a76bf2fc6/init-config-reloader/0.log" Nov 01 22:08:01.525620 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:01.525587 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7v6xv_8b05d35e-918b-4b8c-9643-78f8dff0b109/prometheus-operator/0.log" Nov 01 22:08:01.932498 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:01.932417 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7v6xv_8b05d35e-918b-4b8c-9643-78f8dff0b109/kube-rbac-proxy/0.log" Nov 01 22:08:02.782630 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:02.782587 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9f57576c8-qm8gx_4d11ea05-bc33-4bbc-bcc3-942295c96058/telemeter-client/0.log" Nov 01 22:08:02.848310 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:02.848266 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cf654df49-4ch47_f1ef9e76-0e8b-4050-b269-5042540f7826/authorino/0.log" Nov 01 22:08:03.167580 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:03.167496 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/1.log" Nov 01 22:08:03.168052 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:03.168033 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mlhns_e704eaa9-50e3-4234-9357-f374ad06a72a/manager/2.log" Nov 01 22:08:03.199721 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:03.199695 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9f57576c8-qm8gx_4d11ea05-bc33-4bbc-bcc3-942295c96058/reload/0.log" Nov 01 22:08:03.616096 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:03.616067 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9f57576c8-qm8gx_4d11ea05-bc33-4bbc-bcc3-942295c96058/kube-rbac-proxy/0.log" Nov 01 22:08:04.150068 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:04.150032 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ld4qw_f956c89a-217e-4237-a2cb-70632240750d/limitador/0.log" Nov 01 22:08:04.516912 ip-10-0-128-78 kubenswrapper[5633]: I1101 22:08:04.516874 5633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wdb8s_a8e46f26-efa8-4823-b684-b805cce7bf88/manager/0.log"